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FOSTERING QUALITY SCIENCE AT EPA: 
PERSPECTIVES ON COMMON SENSE REFORM 

(PART I) 


WEDNESDAY, NOVEMBER 30, 2011 

House of Representatives, 
Subcommittee on Energy and Environment, 
Committee on Science, Space, and Technology, 

Washington, D.C. 

The Subcommittee met, pursuant to call, at 2:13 p.m., in Room 
2318 of the Rayburn House Office Building, Hon. Andy Harris 
[Chairman of the Subcommittee] presiding. 


( 1 ) 
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COMMITTEE ON SCIENCE AND TECHNOLOGY 
SUBCOMMITTEE ON ENERGY AND ENVIRONMENT 
U.S. HOUSE OF REPRESENTATIVES 

HEARING CHARTER 


Fostering Quality Science at EPA: Perspectives on Common Sense Reform 


Wednesday, November 30, 201 1 
2:00 p.m. to 4:00 p.m. 

2318 Rayburn House Office Building 


PURPOSE 


On Wednesday, November 30, 201 1, the Subcommittee on Energy and Environment of the Committee 
on Science, Space, and Technology will hold a hearing to provide external perspectives on the need to 
reauthorize and reform science, research and development activities at the Environmental Protection 
Agency (EPA); explore the intersection of Agcncy-supportcd science and its regulatory mission; and 
receive focused recommendations to raise the level, quality, usefulness, and objectivity of EPA science, 
including any necessary changes to the Environmental Research, Development and Demonstration 
Authorization Act. 

WITNESSES 


Ms. Susan Dudley, Director, Regulatory Studies Center, and Research Professor of Public Policy & 
Public Administration, The George Washington University 

Dr. Alan Moghissi, President, Institute for Regulatory Science 

Dr. Kenneth Green, Resident Scholar, American Enterprise Institute 

Dr. Gary Marchant, Professor of Law and Executive Director, Center for Law, Science & Innovation, 
Arizona State University 


BACKGROUND 


The Environmental Research, Development, and Demonstration Authorization Act (ERDDA) authorizes 
research and scientific activities at the Environmental Protection Agency (EPA). Originally enacted in 
1976, Congress subsequently passed annual authorizations through fiscal year 1981. In addition to 
establishing annual authorization levels, these statutes also directed EPA policy in a variety of areas, 
including establishing the Office of Research and Development (ORD)', requiring a 5-year 
environmental R&D plan, and creating EPA’s Science Advisory Board (SAB). 


See Appendix 1 for EPA organizational stnieture. 
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Year 

Act 

Public Law Number 

1976 

ERDDA 

94-475 

1977 

ERDDA of 1978 

95-155 

1978 

ERDDA of 1979 

95-477 

1979 

ERDDA of 1980 

96-229 

1980 

ERDDA of 1981 

96-569 


Since 1981, there have been a number ofbills introduced to reauthorize ERDDA that were not 
ultimately enacted into law.^ As a result, explicit authorization of EPA’s environmental R&D ended at 
the end of fiscal year 1981. This failure to comprehensively reauthorize EPA research, development, 
and demonstration programs and activities illustrates a broader trend among expired environmental 
statutes. The Congressional Research Service notes this trend, stating “Although Congress somewhat 
recently has renewed the authorization of appropriations for certain EPA programs and activities 
through targeted amendments to various statutes, a more comprehensive reauthorization of many of the 
statutes that EPA administers has not been enacted for a number of years. 

In addition to ERDDA, EPA also derives authority for R&D activities through other major 
environmental statutes. For example, under the Clean Air Act, the EPA Administrator must issue 
criteria that “accurately reflect the latest scientific knowledge useful in indicating the kind of extent of 
all identifiable effects on public health or welfare which may be expected from the presence of such 
pollutant in the ambient air.”'^ Through the Safe Drinking Water Act (SDWA), EPA sets standards 
based on “the best available, peer-reviewed science and supporting studies conducted in accordance with 
sound and objective scientific practices.”^ Similarly, the Clean Water Act (CWA) requires EPA to 
publish water quality infonnation “accurately reflecting the latest scientific knowledge.”^ 

In many cases, these major regulatory statutes also authorize specific R&D programs and activities. For 
example, the Clear Air Act established a national research and development program for the prevention 
and control of air pollution including establishing technical advisory committees and research on air 
pollutant monitoring. The SDWA authorized the Administrator of EPA to conduct research and studies 
relating to the causes, diagnosis, treatment, control, and prevention of physical or mental diseases 
resulting directly or indirectly from contaminants in the water including improved methods to identify 
and measure contaminants in drinking water and improved methods to identify and measure the health 
effects of contaminants in drinking water. The CWA directed the Administrator to establish national 
programs for the prevention, reduction, and elimination of pollution and as part of such programs to 
work in cooperation with other State and Federal agencies to coordinate and accelerate research, 


-HR 31 15 (1982), HR 2804 (1982), S. 1205 (1982), S. 2577 (1983), HR 2899 (1984), S. 1292 (1984), HR 2319 (1985), 

S. 2702 (1985), S. 1 144 (1986). HR 2355 (1987), HR 1523 (1987), HR 2153 (1989), HR 4873 (1990), HR 2404 (1991), S. 
1655 (1991), HR 1994 (1993), S, 1545 (1993), HR 2405 (1995), HR 18 14 ( 1995), HR 3322 (1996), HR 1276 (1997), HR 
1742 (1999), HR 1743 (1999). 

Congressional Research Service, “Environinental Laws; Summaries of Major Statutes Administered by the Environmental 
Protection Agency,” RL30798, August 1 1, 201 1. 

42 U.S.C. §7408 (a)(2) (2000). 

^2 U.S.C. §300g-l(b)(3)(A)(i). 

'’33 U.S.C. §1314 (a)(1). 
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investigation, experiments, demonstrations and studies relating to the causes, effects, extent, prevention, 
reduction and elimination of pollution in the navigable waters of the U.S. 

The science enterprise at EPA is spread across program offices and regions. ORD is organized into 
three national labs (comprised of 18 separate labs) and four national centers (which have 19 divisions).^ 
In addition to 1 8 labs within ORD, there are 9 labs split among several program offices and each of the 
10 regions has its own lab.^ In FY2010, the appropriations level for EPA Science and Technology 
activities (S&T includes ORD and the other 19 labs) was $874.9 million. The appropriations level for 
FY201 1 was $840.3 million. The FY20I2 House Committee-passed appropriations level is $777.6 
million and the FY2012 Senate Committee draft appropriations level is $809 million. 

The fragmented nature of EPA R&D presents a challenge to program management and coordination, 
and has complicated efforts to evaluate the effectiveness of these activities. Numerous studies 
conducted by the EPA Office of Inspector General, the Government Accountability Office, and others 
have cited significant concerns with the science activities of the Agency and the difficulties in 
evaluating the usefulness of the science to program needs. These studies have offered recommendations 
on how to improve the science enterprise at EPA, but many of these recommendations have not been 
implemented. 


See Appendix 2. 
Sec Appendix 3. 
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The bottom of this chart shows three national iabs and four national centers. The three national labs are broken down in 
Appendix 3, 
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Chairman Harris. The Subcommittee on Energy and Environ- 
ment will come to order. Good afternoon. Welcome to today’s hear- 
ing entitled “Eostering Quality Science at EPA: Perspectives on 
Commonsense Reform.” In front of you are packets containing the 
written testimony, biographies, and truth in testimony disclosures 
for today’s witness panel. 

I now recognize myself for five minutes for an opening statement. 

I want to welcome everyone to this afternoon’s hearing. This is 
the second in a series of hearings this Subcommittee will be con- 
ducting to provide ideas and guidance to reform science at EPA. 
Unfortunately, the Environmental Research, Development, and 
Demonstration Authorization Act or ERDDA, which is the statute 
authorizing R&D at EPA, as well as the Science Advisory Board, 
was last reauthorized for fiscal year 1981. Thirty years of Congres- 
sional neglect and the aggressive and unjustified regulatory train 
wreck being pursued by this Administration make the time right 
to evaluate reforms to environmental science at the agency. 

Many things have changed since 1981 that demand renewed 
Congressional attention. Eunds appropriated to EPA science and 
technology account have more than tripled from 1981 to 2010, and 
the agency’s overall budget has ballooned to almost $9 billion. The 
agency now employs almost 18,000 people and maintains nearly 40 
laboratories. 

According to the Office of Management and Budget the overall 
effect of all major federal regulations in 1981, were a net cost sav- 
ings of $1 billion. In contrast, in 2010, EPA’s major rules alone rep- 
resented over 23 billion in costs, a figure itself that many believe 
is a significant underestimate, and there have been disagreements 
as to the real benefits of these regulations. 

There are also very pragmatic reasons for us to be keenly inter- 
ested in reforming and reauthorizing science activities at the EPA. 
Given the dire fiscal straits that our country is facing, programs, 
activities, and agencies that are operating under expired or out- 
dated authorizations will have targets on their backs as we seek 
to get our budgetary house in order. 

In light of this the right reforms to EPA R&D programs will not 
only improve trust in the science that informs regulatory decisions 
but will also provide a framework to prioritize the most important 
functions and reduce unnecessary and wasteful spending else- 
where. 

Eor instance, despite 1.2 million examples of successful hydrau- 
lically-fractured wells, the agency is moving forward with an un- 
necessary study in the area. 

Some basic questions need to be asked. What should the role of 
EPA be in conducting research? Should it be limited to funda- 
mental research? Should R&D be limited to supporting the agency’s 
regulatory agenda? What is the relationship between EPA’s science 
and policymaking missions? And how do we prevent the politicizing 
of scientific activities? How can Congress best ensure regulatory 
science that is reliable, peer-reviewed, transparent, understand- 
able, and objective? Are structural changes necessary to improve 
the quality and independence of the agency’s scientific advisory 
bodies? And do we have our environmental priorities right? And 
are we getting the most environmental bang for our buck? 
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This hearing follows up on testimony received two weeks ago 
from officials at the EPA’s Office of Research and Development, Of- 
fice of the Inspector General, and the Government Accountability 
Office. 

Furthermore, in order to build a substantive record this is actu- 
ally the ninth hearing on science and process at the EPA that this 
Committee has held so far in the 112th Congress. The Committee 
has also sent a series of letters to EPA and the Administration re- 
questing further information about policies on transparency, cost 
benefit analysis, and peer review. Unfortunately, we are still wait- 
ing for responses to four letters sent since September. 

Reforming environmental science should not be a partisan issue, 
as the 2009 report by the Bipartisan Policy Center’s Science for 
Policy Project co-chaired by the former chair of the full Science 
Committee, Sherry Boehlert, explained, “A tendency to frame regu- 
latory issues as debates solely about science, regardless of the ac- 
tual subject in dispute, is at the root of the stalemate and acrimony 
all too present in the regulatory system today.” 

The report went on to recommend that Congress should include 
their recommendations, “In legislation as relevant programs are re- 
authorized,” including suggesting the studies used in developing 
regulations should be subject to data access requirements, agencies 
and advisory bodies should be transparent in their approach to 
evaluating weighing studies, and that agencies should explicitly 
differentiate between scientific judgments and policy judgments. 

These are reasonable core principles that I hope both sides can 
agree upon and which will advance fulfillment of the President’s 
executive order requiring that, “Our regulatory system must be 
based on the best available science.” 

The diverse set of witnesses with us today will offer their views 
on these and other EPA scientific reform ideas and offer rec- 
ommendations for improving and clarifying environmental R&D 
priorities. I hope these suggestions will highlight some potential 
avenues for bipartisan cooperation as our Subcommittee continues 
its work on these issues. 

Again, I want to thank all the witnesses for appearing before the 
Subcommittee, and I look forward to a constructive discussion. 

[The prepared statement of Mr. Harris follows:] 

Prepared Statement of Chairman Andy Harris 

I want to welcome everyone to this afternoon’s hearing on Fostering Quality 
Science at EPA: Perspectives on Common Sense Reform. 

This is the second in a series of hearings this Subcommittee will he conducting 
to provide ideas and guidance to reform science at EPA. Unfortunately, the Environ- 
mental Research, Development and Demonstration Authorization Act, or ERDDAA 
(ERDDA), which is the statute authorizing R&D at EPA as well as the Science Advi- 
sory Board, was last reauthorized for fiscal year 1981. Thirty years of Congressional 
neglect and the aggressive and unjustified regulatory train wreck being pursued by 
this administration make the time ripe to evaluate reforms to environmental science 
at the Agency. 

Many things have changed since 1981 that demand renewed Congressional atten- 
tion. Funds appropriated to EPA’s science and technology account have more than 
tripled from 1981 to 2010, and the Agency’s overall budget has ballooned to almost 
$9 billion dollars. The Agency now employs almost 18,000 people, and maintains 
nearly 40 laboratories. According to the Office of Management and Budget, the over- 
all effect of all major federal regulations in 1981 was a net cost savings of $1 billion. 
In contrast, in 2010, EPA’s major rules alone represented over $23 billion in costs — 
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a figure itself that many believe is a significant underestimate and there have been 
disagreements as to the real benefits of these regulations. 

There are also very pragmatic reasons for us to be keenly interested in reforming 
and reauthorizing science activities at the EPA. Given the dire fiscal straits that 
our country is facing, programs, activities, and agencies that are operating under 
expired or outdated authorizations will have targets on their backs as we seek to 
get our budgetary house in order. In light of this, the right reforms to EPA R&D 
programs will not only improve trust in the science that informs regulatory deci- 
sions, it will also provide a framework to prioritize the most important functions 
and reduce unnecessary and wasteful spending elsewhere. For instance, despite 1.2 
million examples of successful hydraulically-fractured wells, the Agency is moving 
forward with an unnecessary study in this area. 

Some basic questions need to be asked: What should be the role of EPA in con- 
ducting research? Should it be limited to fundamental research? Should R&D be 
limited to supporting the Agency’s regulatory agenda? What is the relationship be- 
tween EPA’s science and policymaking mission, and how do we prevent the politi- 
cizing of scientific activities? How can Congress best ensure regulatory science that 
is reliable, peer reviewed, transparent, understandable, and objective? Are struc- 
tural changes necessa^ to improve the quality and independence of the Agency’s 
scientific advisory bodies? And do we have our environmental priorities right, and 
are we getting the most environmental bang-for-our-buck? 

This hearing follows up on testimony received two weeks ago from officials from 
EPA’s Office of Research and Development, Office of the Inspector General, and the 
Government Accountability Office. Furthermore, in order to build a substantive 
record, this is actually the ninth hearing on science and process at the Environ- 
mental Protection Agency that this Committee has held so far in the 112th Con- 
gress. The Committee has also sent a series of letters to EPA and the Administra- 
tion requesting further information about policies on transparency, cost-benefit 
analysis, and peer review. Unfortunately, we are still waiting for responses to four 
letters sent since September. 

Reforming environmental science should not be a partisan issue. As a 2009 report 
by the Bipartisan Policy Center’s Science for Policy Project, co-chaired by the former 
Chair of the full Science Committee, Sherry Boehlert, explained: “A tendency to 
frame re^latory issues as debates solely about science, regardless of the actual sub- 
ject in dispute, is at the root of the stalemate and acrimony all too present in the 
regulatory system today.” The report went on to recommend that Congress should 
include their recommendations “in legislation as relevant programs are reauthor- 
ized,” including suggesting that studies used in developing regulations should be 
subject to data access requirements, agencies and advisory bodies should be trans- 
parent in their approach to evaluating and weighing studies, and that agencies 
should explicitly differentiate between scientific judgments and policy judgments. 

These are reasonable core principles that I hope both sides can agree on and 
which will advance fulfillment of the President’s executive order requiring that “Our 
regulatory system . . . must be based on the hest available science.” 

The diverse set of witnesses with us today will offer their views on these and 
other EPA scientific reform ideas, and offer recommendations for improving and 
clarifying environmental R&D priorities. I hope these suggestions will highlight 
some potential avenues for bipartisan cooperation as our Subcommittee continues its 
work on these issues. 

I want to thank the witnesses for appearing before the Subcommittee and I look 
forward to a constructive discussion. 

Chairman Harris. The chair now recognizes Mr. Miller, the 
Ranking Member, for five minutes for an opening statement. 

Mr. Miller. Thank you, Mr. Chairman. I appreciate your accom- 
modating me by beginning slightly early this afternoon to allow me 
to cast votes in the Financial Services Committee, and I would also 
like to make this Committee’s efforts in rewriting the authorization 
statute to be a bipartisan effort. I appreciate your willingness to 
work with us to try to make future hearings more useful to the 
Committee in informing that important work. 

Today the Subcommittee meets again for part two of the EPA re- 
search and science, series of hearings. The first hearing two weeks 
ago was disappointing and a missed opportunity. The stated pur- 
pose of the hearing a couple weeks ago was to examine the ability 
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of EPA’s research enterprise to meet the agency’s mission to pro- 
tect public health and the environment. 

However, many of my colleagues decided instead to use their 
time to focus on EPA’s hydraulic fracturing study rather than on 
that stated purpose. I believe the goal of this series of hearings 
should be to establish a useful Committee record to prepare the 
right legislation to reauthorize the Environmental Research, Devel- 
opment, and Demonstration Authorization Act, ERDDA. 

Today’s hearing does not appear to be any more likely to inform 
the Committee about structural and substantive concerns of stake- 
holders related to EPA’s research activities, and it is not a bal- 
anced, comprehensive, or even a helpful hearing. This hearing 
looks like what we are seeing on the House Eloor this week, a plat- 
form for anti-regulation, anti-science talking points, and as I said 
a couple of weeks ago, I hope that my Republican counterparts are 
truly interested in reform that will lead to better research to en- 
hance public health and protect the environment. 

Although we all agree that there are legitimate concerns related 
to EPA’s research enterprise, this hearing doesn’t really help us 
understand or address those issues. The agency’s scientific research 
is important as more complex environmental issues emerge and 
evolve that need to be understood and addressed. Scientific re- 
search knowledge and technical information are fundamental to 
EPA’s mission and inform its standard setting, regulatory compli- 
ance, and enforcement functions. That is why Congress saw fit to 
create advisory bodies at EPA like the Clean Air Scientific Advi- 
sory Committee, which was created to provide independent advice 
on the science and allow the Administrator to make regulatory de- 
cisions. 

I really hope today that we would have a productive conversation 
about how best to position EPA to perform that mission, protecting 
human health and the environment. And if we are really serious 
about working towards reauthorizing ERDDA, which does need to 
be done as the Chairman said, after 30 years perhaps it is wise to 
revisit that statute, we must establish a Committee record that will 
offer us a wide range of views on how best to draft legislation that 
would serve the agency better, as well as the people that we all 
represent. 

If that is the case, I hope that we can commit to working to- 
gether after today to put together hearings and panels that will 
serve that purpose. 

And with that. Chairman Harris, I yield back. 

[The prepared statement of Mr. Miller follows:] 

Prepared Statement of Ranking Member Brad Miller 

Thank you Chairman Harris. Today the Subcommittee meets again for part two 
of the EPA research and science series of hearings. The first hearing two weeks ago 
was pretty disappointing and a missed opportunity. The stated purpose of our hear- 
ing a couple of weeks ago was to examine the ability of EPA’s research enterprise 
to meet the agency’s mission to protect public health and the environment. However 
many of my colleagues on the other side of the aisle decided to use their time to 
focus on EPA’s hydraulic fracturing study rather than on that stated purpose. I be- 
lieve the goal of this series of hearings is to establish a useful Committee record 
in preparation of writing legislation to reauthorize the Environmental Research, De- 
velopment, and Demonstration Authorization Act (ERDDA). 
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Today’s hearing does not appear to be any more likely to inform the Committee 
about structural and substantive concerns of stakeholders related to EPA’s research 
activities. It is not balanced, comprehensive, or even helpful. This hearing looks just 
like what we are seeing on the House floor this week — the anti-regulation, anti- 
science talking points of the far right. As I said a couple of weeks ago, I hoped that 
my Republican counterparts were really interested in reform that will lead to better 
research to enhance public health and protect the environment. Although we all 
agree that there are legitimate concerns related to EPA’s research enterprise, this 
hearing doesn’t come close to helping us understand or address these issues. The 
agency’s scientific research is important as more complex environmental issues 
emerge and evolve that need to be understood and addressed. Scientific research, 
knowledge, and technical information are fundamental to EPA’s mission and inform 
its standard-setting, regulatory, compliance, and enforcement functions. That is why 
Congress saw fit to create advisory bodies at EPA, such as the Clean Air Scientific 
Advisory Committee (CASAC), which was created to provide independent advice on 
the science which allows the Administrator to make regulatory decisions. 

I really hoped that today we would have a productive conversation about how best 
to position the EPA to perform its mission of protecting human health and the envi- 
ronment. If we are really serious about working towards reauthorizing ERDDA, we 
must establish a Committee record that will offer us a wide-range of views on how 
best to draft legislation to better serve the agency as well as the people that we 
all represent. If that is the case, I hope that we can commit to working together 
after today in formulating hearings and panels that will serve that purpose. 

With that. Chairman Harris, I 3deld back. 

Chairman Harris. Thank you very much, Mr. Miller. 

If there are Members who wish to submit additional opening 
statements, your statements will be added to the record at this 
point. 

At this time I would like to introduce our witness panel. Again, 
I want to thank you for your patience. For the slightly delayed 
start. Our first witness today is the Honorable Susan Dudley, Di- 
rector of the Regulatory Studies Center and Research Professor of 
Public Policy and Public Administration at the George Washington 
University. From April, 2007 through January, 2009, Professor 
Dudley served as the Presidentially-appointed Administrator of the 
Office of Information and Regulatory Affairs in the U.S. Office of 
Management and Budget. 

The next witness will be Dr. Alan Moghissi, the President of the 
Institute for Regulatory Science. Previously working for EPA Dr. 
Moghissi managed numerous programs, including the Bio-Environ- 
mental Radiological Program at the National Environmental Re- 
search Center in Las Vegas, the Health and Environmental Risk 
Analysis Program in Washington, DC. He was also the Principle 
Science Advisor for Radiation and Hazardous Materials and rep- 
resented the Office of Research and Development in a number of 
work groups responsible for drafting environmental regulations. 

The next witness is Dr. Kenneth Green, a Resident Scholar at 
the American Enterprise Institute. Dr. Green has studied public 
policy and regulation at Free Enterprise Think Tanks across North 
America for nearly 20 years. An environmental scientist by train- 
ing, Dr. Green focuses on policy and regulations involving energy 
and environmental health. 

And the final witness today is Dr. Gary Marchant, Professor of 
Law and Executive Director of the Center for Law, Science, and In- 
novation at Arizona State University. Prior to joining the ASU fac- 
ulty in 1999, he was a partner in the Washington, DC, law firm, 
Kirkland & Ellis, where his practice focused on regulatory issues. 
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Thank you all for appearing before the Subcommittee today. As 
our witnesses should know, spoken testimony is limited to five min- 
utes each, after which the Members of the Committee will have five 
minutes each to ask questions. 

I now recognize our first witness, Ms. Susan Dudley, Director of 
the Regulatory Studies Center and Research Professor of Public 
Policy and Public Administration at the George Washington Uni- 
versity. 

Ms. Dudley. 

STATEMENT OF MS. SUSAN DUDLEY, DIRECTOR, 
REGULATORY STUDIES CENTER, AND RESEARCH PROFESSOR 
OF PUBLIC POLICY AND PUBLIC ADMINISTRATION, 
GEORGE WASHINGTON UNIVERSITY 

Ms. Dudley. Thank you. Chairman Harris, Ranking Member 
Miller, and Members of the Subcommittee. As you mentioned I am 
at G.W. but my remarks here today are my own. 

EPA regulations are often the subject of heated debate involving 
accusations of politicized science and advocacy science. While it is 
legitimate to be wary of policy officials and politicians trying to in- 
fluence scientific studies, more often than not, these debates center 
on issues that science can inform but not decide. 

And I am actually going to read a quote from the Bipartisan Pol- 
icy Center Report that the chairman mentioned. In fact, you read 
the same quote, but I am going to repeat it again. This is the Bi- 
partisan Policy Center Report, “Improving the Use of Science and 
Regulatory Policy.” “ A tendency to frame regulatory issues as de- 
bates solely about science, regardless of the actual subject in dis- 
pute, is at the root of the stalemate and acrimony all too present 
in the regulatory system today.” 

And I do highly recommend that report. I think it does show that 
these issues are not partisan. Framing issues as debates solely 
about science is problematic for two reasons. 

First, while science is essential for understanding the positive 
question of what is, it is less helpful for the normative policy ques- 
tion of what should be. Sound policy decisions depend not only on 
scientific assessments of risk but also on other factors such as eco- 
nomics, ethics, law, and, yes, politics, the will of the people. 

Second, scientists will never have complete information to predict 
outcomes with absolute certainty, so even the risk assessment as 
opposed to the risk management phase of an analysis depend on 
assumptions and judgments that guide the use of scientific infor- 
mation. 

Policymakers and the public are often unaware of the influence 
of these risk assessment policy choices or the existence of alter- 
native assessments that are equally plausible. Instead, assess- 
ments often generate precise-sounding predictions that hide consid- 
erable uncertainty about the actual risk and heavily are influenced 
by hidden judgments about what policies should look like. 

Institutional arrangements in the regulatory development proc- 
ess tend to aggravate these problems, perpetuating the charade 
that policies are based purely on science, insulating experts in- 
volved in a particular rulemaking from dissenting views, rein- 
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forcing preconceptions and biases, and leading to regulatory policy 
decisions that are not at all transparent. 

As the Committee evaluates approaches to address perceived 
problems in the quality of EPA science, it is important to identify 
whether the source of the problem is politicians attempting to con- 
trol science or the politicization of science, or scientists attempting 
to control policy, something that David Goldston, who was the Ex- 
ecutive Director of the Bipartisan Policy Center Project on this and 
now at NRDC, calls that the scientification of policy. 

My own experience supports the Bipartisan Policy Center’s con- 
clusion that the latter problem is behind much of the controversy 
related to science-based regulation. So with that in mind, let me 
offer three modest recommendations. 

One, recognize that science is a positive discipline that can in- 
form but not decide appropriate policy. Avoid the temptation to del- 
egate decisions to agencies on the pretense that science alone can 
make the normative decision of what the policy should be. I was 
going to quote again from the Bipartisan Policy Center report, but 
I won’t in the interest of time. 

My second recommendation to recognize that risk assessment 
necessarily involves assumptions and judgments as well as pure 
scientific inputs and establish procedures and incentives to make 
more transparent risk assessment inputs in the range of plausible 
outcomes. 

The National Academies of Science has offered numerous rec- 
ommendations in this regard for improving the quality and trans- 
parency of the EPA risk assessment, including in its recent report 
earlier this year on the formaldehyde IRIS process. 

And my final recommendation is to increase the robustness of 
regulatory science by institutionalizing feedback mechanisms along 
with checks and balances. The scientific method depends on 
falsifiable hypotheses, data gathering, dissent, and challenge to en- 
sure objective analysis to minimize bias in the interpretation of re- 
sults. 

Now, no one is truly objective, so institutional reforms that en- 
gage and encourage competing views could go a long way to im- 
prove the clarity of the risk assessment process and the decisions 
that depend on scientific input. 

President Obama has taken some positive steps in this regard, 
reinforcing interagency review and calling for more open exchange 
with the public. 

Other successful reforms might involve pre-rulemaking disclo- 
sure of risk assessment information to engage broad public com- 
ment on the proper choice of studies, models, assumptions, et 
cetera, long before any policy decisions are framed and positions 
are set in stone. 

And with 1 second left I will stop. Thank you. 

[The prepared statement of Ms. Dudley follows:] 

Prepared Statement of Ms. Susan Dudley, Director, 

Regulatory Studies Center, and Research Professor of Public Policy 
& Public Administration, The George Washington University 

Chairman Harris, Ranking Member Miller, and Members of the Subcommittee, 
thank you for inviting me to testify today on “Fostering Quality Science at EPA: 
Perspectives on Common Sense Reforms.” I am Director of the George Washington 
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University Regulatory Studies Center and Research Professor in the Trachtenberg 
School of Public Policy and Public Administration. ^ 

From April 2007 to January 2009, I oversaw the executive branch regulations of 
the federal government as Administrator of the Office of Information and Regu- 
latory Affairs (OIRA) in the Office of Management and Budget (0MB). I have de- 
voted my career to trying to improve both the framework for developing regulations 
and our understanding of regulations’ effects, and for over three decades have exam- 
ined regulations from perspectives in government (as both a career civil servant and 
political appointee), academia, consulting, and the non-profit world. 

EPA regulations intended to address public health and environmental risks de- 
pend on scientific information. They are often the subject of heated debate involving 
accusations of “politicized science” and “advocacy science,” as everyone — including 
scientists and agency officials — wields scientific information in the service of advo- 
cacy. While it is legitimate to be wary of politicians or policy officials trying to influ- 
ence scientific studies, more often than not, these debates center on issues that 
science can inform, but not decide. 

As the Bipartisan Policy Center, in its 2009 report. Improving the Use of Science 
in Regulatory Policy, observed: 

Political decision-makers should never dictate what scientific studies should con- 
clude, and they should base policy on a thorough review of all relevant research 
and the provisions of the relevant statutes. But some disputes over the 
“politicization” of science actually arise over differences about policy choices that 
science can inform, but not determine. (BPC 2009, 4) 

Science is rarely sufficient for making policy decisions for two reasons. First, 
while science is essential for understanding the positive question of what is, or pre- 
dicting what outcomes might derive under different scenarios, it is less helpful for 
the normative (policy) decisions regarding what should be. Sound policy decisions 
depend not only on scientific assessments of risk, but also on other factors, such as 
economics, ethics, law, and politics — the will of the people. 

Second, scientists will never have complete information to predict outcomes with 
absolute certainty, so risk assessors use what the National Research Council (NRC 
1983) called “risk assessment policy” — assumptions and rules of thumb — to guide 
the use of scientific information in analyses that inform policy in the face of uncer- 
tainty. 

In each step [of the risk assessment process], a number of decision points (compo- 
nents) occur where risk to human health can only be inferred from the available 
evidence. Both scientific judgments and policy choices may be involved in selecting 
from among possible inferential bridges, and we have used the term risk assess- 
ment policy to differentiate those judgments and choices from the broader social 
and economic policy issues that are inherent in risk management decisions. (NRC) 

Policymakers and the public are often unaware of the influence of these risk as- 
sessment policy choices or the existence of alternative assessments that are equally 
plausible. Instead, assessments often generate precise-sounding predictions that 
hide considerable uncertainty about the actual risk. Since EPA’s stated policy is to 
err on the side of overstating risk, it relies on one-sided policy choices at each node 
in the risk assessment process. Policy decisions that are reported as if they are 
based on science are heavily are influenced by these hidden staff judgments about 
what policies should be. 

While some judgment is necessary to translate scientific evidence into risk assess- 
ment, current risk assessment policies lead to distortions in risk estimates and false 
precision in the presentation of scientific information. This threatens the scientific 
credibility of the process, hiding rather than making transparent the uncertainty in 
assessments of risk, putting key policy choices in the hands of staff, and allowing 
policy makers to avoid making hard decisions. 

when questions involving policy judgment and values are falsely characterized as 
scientific, a small number of people have an effective monopoly on the information 
that is used and how it is characterized, leading to decisions that are not as ac- 
countable or as transparent as they should be. “When regulators purport to rely on 


^The George Washington University Regulatory Studies Center raises awareness of regula- 
tions’ effects with the goal of improving regulatory policy through research, education, and out- 
reach. This statement reflects my views, and does not represent an official position of the GW 
Regulatory Studies Center or the George Washington University. 
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science as the sole basis for their policy choices, the real reasons justif 3 dng their 
choices remain hidden from public view.” (Coglianese 2009) This is exacerbated by 
the adversarial nature of rulemaking, and group dynamics that discourage dif- 
ferences of opinion and lead to poor decisions that mask uncertainty and give short 
shrift to important factors and perspectives. 

Institutional arrangements in the regulatory development process tend to aggra- 
vate these problems, perpetuating the charade that policies are based purely on 
science, insulating experts involved in a particular rulemaking from dissenting 
views, reinforcing preconceptions and biases, and leading to regulatory policy deci- 
sions that are not at all transparent. 

Statutory mandates, such as those directing EPA to set National Ambient Air 
Quality Standards (NAAQS) for “criteria pollutants” under the Clean Air Act, can 
make inevitable the “science charade,” where regulatory agencies “camouflagle] con- 
troversial policy decisions as science.” (Wagner 1995, 1614) Congress directs EPA 
to set NAAQS at a level that is “requisite to protect public health with an adequate 
margin of safety,” but restricts the agency from considering key factors, establishing 
instead the pretense that science is sufficient to determine a single point concentra- 
tion that is “requisite to protect public health.” The courts have reinforced a limited 
interpretation of the Act, as well as tight deadlines for issuing revised standards. 
Executive branch career and policy officials respond by developing scientific-sound- 
ing explanations to justify one standard over another. Analysts have an incentive 
to downplay rather than reveal the implications of key risk assessment policy 
choices, and decision makers point to science as either requiring a new standard or 
as being so uncertain that a new standard cannot be set. The interagency review 
process is often truncated by very short timeframes established by the statute and 
reviewing courts, and constrained by the limited range of options presented by EPA 
and its Clean Air Science Advisory Committee. Public interveners vigorously defend 
alternative standards based on their own interpretation of the science. 

This has evolved into an adversarial process, characterized by harsh rhetoric in 
which each party claims the science supports its recommended policy outcome and 
questions opponents’ credibility and motives, rather than a constructive discussion 
regarding appropriate assumptions and data and the reasonableness of the statu- 
tory goal. The real reasons for selecting a non-zero standard are not transparent. 

As the Subcommittee evaluates approaches to address perceived problems with 
the “quality, usefulness and objectivity of EPA science,” it is important to identify 
whether the source of the problem is: 

A. politicians attempting to control science (“politicization of science”), or 

B. scientists attempting to control policy (“scientification of policy.”) 

My own experience supports the BPC conclusion that this latter problem is behind 
much of the controversy related to science-based regulation, and is the main contrib- 
utor to the science charade: 

A tendency to frame regulatory issues as debates solely about science, regardless 
of the actual subject in dispute, is at the root of the stalemate and acrimony all 
too present in the regulatory system today. (BPC 2009, 10) 

Current procedures for developing regulations addressing health and environ- 
mental risk blur the lines between science and policy, hindering not only public pol- 
icy decisions, but development of scientific knowledge itself. Current institutions 
provide incentives to bury policy judgments in analyses that are presented as 
science, perpetuating the science charade. 

Altering these incentives is challenging, and I appreciate this Subcommittee’s in- 
terest in this subject. In a chapter of a forthcoming book, ^ my coauthor Professor 
George Gray and I offer modest suggestions aimed at increasing transparency in 
regulatory science, strengthening the checks and balances provided by different par- 
ticipants in the rulemaking process, and engaging a broad range of expertise and 
perspectives to counter the problems insular decision-making brings. Those sugges- 
tions are the basis for a few recommendations to the Subcommittee. 

1. Recognize that “science” is a positive discipline that can inform, but not decide, 
appropriate policy. Avoid the temptation to delegate decisions to agencies on the 


2 Institutions and Incentives in Regulatory Science, Lexington Books, Jason Johnston ed., 
forthcoming spring 2012. 
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pretense that “science” alone can make the normative determination of what 
policy should be. 

The BPC observed: 

The first impulse of those concerned with regulatory policy should not be to claim 
“the science made me do it” or to dismiss or discount scientific results, but rather 
to publicly discuss the policies and values that legitimately affect how science gets 
applied in decision making. (BPC 2009, 4) 

Distinguishing between science and policy is not always easy or straightforward, 
and scientists may make choices based on values in the course of their work. None- 
theless, policy debate would be clarified and enhanced if a systematic effort were 
made to distinguish between questions that can be resolved through scientific judg- 
ments and those that involve judgments about values and other matters of policy 
when regulatory issues comprise both. This transparency would both help force val- 
ues debates into the open and could limit spurious claims about, and attacks on 
science. (BPC 2009, 15) 

Legislators should also take care to limit the role of scientific advisory panels to 
advising on science, and not to embed their policy views in their scientific rec- 
ommendations. The BPC recommended: 

In general, scientific advisory panels should not be asked to recommend specific 
regulatory policies. (BPC 2009, 5) 

2. Recognize that risk assessment necessarily involves assumptions and judgments 
as well as pure scientific inputs, and establish procedures and incentives to 
make more transparent risk assessment inputs and the range of plausible out- 
comes. 

Efforts to identify and characterize the uncertainty in scientific evidence by quan- 
tifying the range of outcomes of potential regulatory actions may provide useful data 
for improving risk assessment policy choices and increasing confidence in decisions. 

The BPC recommended: 

In presenting the conclusions of literature reviews, agencies and their scientific 
advisory committees need to be as open and precise as possible in discussing levels 
of risk and uncertainty. Policy makers should be wary of conclusions about risk that 
are expressed as a single number. (BPC 2009, 8) 

3. Increase the robustness of regulatory science by institutionalizing feedback 
mechanisms, checks, and balances. 

Greater transparency in the models, assumptions, and risk assessment policy 
choices could encourage more open, constructive debate on those choices. The sci- 
entific method depends on falsifiable hypotheses, data gathering, dissent, and chal- 
lenge to ensure objective analysis to minimize bias in the interpretation of results. 

No one is truly objective. We all approach problems with our own “priors” and, 
particularly when faced with new or incomplete information, we tend to look to oth- 
ers in whom we trust to help form our opinions and make decisions. Cass Sunstein’s 
interesting research on “why groups go to extremes” shows that individuals form 
more extreme views when surrounded by others with similar perspectives. Institu- 
tional reforms that engage competing views could go a long way to improve the clar- 
ity of the risk assessment process and the decisions that depend on scientific input. 

President Obama has built on his predecessors’ efforts to provide for interagency 
review of different aspects of regulatory decisions, including the underl 3 dng science. 
He has directed agencies to encourage an “open exchange of information and per- 
spectives among State, local, and tribal officials, experts in relevant disciplines, af- 
fected stakeholders in the private sector, and tbe public as a whole, including rel- 
evant scientific and technical findings.” 

Successful reforms might involve pre-rulemaking disclosure of risk assessment in- 
formation, to engage broad public comment on the proper choice of studies, models, 
assumptions, etc. long before any policy decisions are framed, and “positions” estab- 
lished. 


I appreciate this Subcommittee’s interest in improving how science informs envi- 
ronmental regulation, and welcome opportunities to discuss the likely effects of dif- 
ferent reforms. 
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Chairman Harris. Thank you very much. 

I now recognize our second witness, Dr. Alan Moghissi, the Presi- 
dent of the Institute for Regulatory Science. 

STATEMENT OF DR. ALAN MOGHISSI, PRESIDENT, INSTITUTE 
FOR REGULATORY SCIENCE 

Dr. Moghissi. Chairman Dr. Harris, Ranking Member Miller, 
Members of the Subcommittee, as the chairman said my name is 
Alan Moghissi, and I am the President of Institute for Regulatory 
Science. We are not for profit organization established in 1985, and 
we are in Alexandria, Virginia. We are dedicated to the idea that 
societal decisions notably environmental regulations must be based 
on what we call best available science, and Mr. Chairman, you 
used that term, and that was music to my ear. 

And we define best available science if you go to our website at 
www.nars.org, you will find upon the description of BAS, my testi- 
mony to you is based on that, however, my — it is — what I am doing 
today is my personal interpretation of BAS rather than official pro- 
nouncement of the Institute for Regulatory Science. 

Although the term regulatory science is used extensively in the 
interest of transparency, let me define it again. Regulatory science 
consists of the scientific foundation of policy, notably regulatory de- 
cisions. It is the science part of the subject rather than societal im- 
plications of the regulatory side. 

Based on this definition the scientific activity of the EPA are 
overwhelmingly regulatory science. I appreciate the opportunity to 
testify before your Committee, and I am proposing the Congress to 
enact the Regulatory Science Sunshine Act, Regulatory Science 
Sunshine Act as a segment of the EPA Authorization Appropriation 
or as a separate act. 

My written testimony includes metrics for evaluation of regu- 
latory science information and regulatory science ethics. My apolo- 
gies that I cannot describe them because I needed more than 1 
hour to do so, and I have only five minutes. 

Please note that the regulatory science information at best in the 
metrics that I included partially reproducible evolving science and 
often at lower maturity going all the way to scadon to scientific 
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judgment or even speculation. A characteristic of the class I just 
described a partially reproducible evolving science is that although 
its foundation is proven science and uncontested, it often uses a 
macolic on the right side described, assumptions, judgment, default 
data, and values if the relevant data or values aren’t all available 
and most unfortunately as Mr. Chairman pointed out, on occasion 
it uses societal adjustments that are outside the purview of science. 

The proposed Regulatory Science Sunshine Act would require 
that within the R&D Program EPA makes a concerted effort to 
double up procedures, processes, and methods for each regulatory 
science, regulatory decision that is based on or includes science, 
one, identification of assumptions, judgment, default data, any 
other similar system used in the regulatory process, identification 
of potential alternatives, and how the conclusions would be dif- 
ferent if an alternative assumption, judgment or similar param- 
eters are used. 

Two, description of the content of all mathematical equations in 
words. 

Three, information identified above must be written in a lan- 
guage that is understandable to knowledgeable non-specialists or 
better yet to an average person. 

Clear and unambiguous justification for the inclusion of societal 
objectives in science rather than addressing societal objectives in 
the administrative decision process. 

And obligation of EPA to comply with ethical requirement of reg- 
ulatory science. 

I am a proud member of the — proud charter member of the EPA 
and believe that EPA has done an outstanding job in protecting 
human health and the environment. However, I would be less than 
frank if I would not express my concern over certain decisions that 
have had adverse societal including environmental consequences. 

There are those in the regulatory science community who believe 
that members of the other scientific disciplines have settled the 
general public without difficulties understanding the complex na- 
ture of regular — of scientific — of regulations. They are wrong. They 
are dead wrong. 

Quoting Sir Thomas Jefferson as did my boss, Ruckelshaus, “If 
we think the people are not enlightened enough to exercise their 
control with a wholesome discretion, the remedy is not to take it 
away from them but to inform their discretion.” 

Thank you much. 

[The prepared statement of Mr. Moghissi follows:] 

Prepared Statement of Dr. Alan Moghissi, President, Institute for 
Regulatory Science 

Chairman Dr. Harris, Ranking Member Miller, and Members of the Sub- 
committee; I am A. Alan Moghissi, President of Institute for Regulatory Science 
(RSI). We were established in 1985 as a not-for-profit organization located in Alex- 
andria, VA and. We are dedicated to the idea that societal decisions notably environ- 
mental regulations must be based on what we call “Best Available Science” or BAS. 
I am a proud charter member of the U.S. Environmental Protection Agency (EPA) 
and believe that EPA has done an outstanding job in protecting human health and 
the environment but I am less proud that EPA has missed some opportunities to 
use BAS in its decisions. I appreciate the opportunity to testify before your Com- 
mittee and intend to suggest that the time has come for the EPA to substantially 
expand transparency in the scientific foundation of its regulatory activities. 
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Science at the EPA and the Establishment of Regulatory Science 

Looking back at the history when EPA was formed, although there were laws 
dealing with air, water, and food, the ability of government to adequately regulate 
emission of toxic agents was limited. For example, there was no law that provided 
government for regulating manufacturing of chemicals. During that period the Con- 
gress quickly passed a number of laws mandating promulgation of regulations at 
a rapid pace. Upon the formation of the EPA, the managers and scientists at that 
Agency were faced with the urgent need to promulgate a large number of regula- 
tions based on deadlines mandated by legislative actions or judicial decisions. This 
problem caused the EPA to rely upon the judgment of scientists, short cutting sci- 
entific issues, and use their best to meet the deadlines. During this initial phase 
of the EPA the phrase regulatory science appeared describing the scientific seg- 
ments or parts of regulations. Meanwhile regulatory science is defined as follows: 

Regulatory science consists of the scientific foundation of policy notably regulatory 
decisions 

Regulatory science, sometimes called regulatory sciences, covers many disciplines 
(Moghissi et al, 2011). It includes regulatory toxicology, regulatory ecology, regu- 
latory hydrology, and regulatory atmospheric sciences, to mention a few. It is no dif- 
ferent than other disciplines such as chemistry discipline that covers, inorganic 
chemistry, organic chemistry, biochemistry, physical chemistry, chemical engineer- 
ing, and medicinal chemistry, to mention a few. 

As expected virtually all regulatory agencies must deal with regulatory science in 
promulgating their regulations. For example the Food and Drug Administration 
(FDA) has not only used that term to describe its scientific objectives but also has 
devoted significant funds for R&D devoted to regulatory science in areas of its regu- 
latory authority. Similarly, the EPA has an extensive regulatory science program 
both in its R&D and program offices, although that term is not always used in its 
pronouncements . 

With the maturity of the EPA’s regulatory process the EPA is provided significant 
funding for R&D. A discussion of relevancy of EPA’s R&D to its mission, the quality 
of science used in its regulatory process and related issues have been addressed nu- 
merous times and most recently, in testimonies before this Committee (Anastas 
2011, Trimble 2011, Elkins 2011). Therefore, this testimony will address the trans- 
parency issue, a subject that appears to have been insufficiently addressed. As stat- 
ed above, during its initial phases of operation, EPA was facing deadlines and had 
to go through shortcuts. Meanwhile, the EPA has time to thoroughly evaluate the 
scientific foundation of its regulations. An example of these regulations is emission 
limits being considered for greenhouse gases. EPA did not face a deadline and based 
on its own desire undertook the laborious and highly contested decision to regulate 
greenhouse gases. 

Subsequent to the formulation of the term regulatory science, my colleagues and 
I tried to develop a systematic process for evaluation of regulatory science informa- 
tion. We had to identify fundamental principles not only for regulatory science but 
also for any scientific claim. We had also to address how does an organization in- 
cluding a regulatory agency assesses the reliability of a scientific claim regardless 
of its origin. We struggled for many years to address the level of maturity of sci- 
entific information. Finally, we had to address the issue of science vs. areas outside 
the purview of science. These efforts took over three decades and have reached suffi- 
cient maturity that can be described here. 

Metrics for Evaluation of Regulatory Science Information 

As stated above, the development of the BAS system and Metrics for Evaluation 
of Regulatory Science Information (MERSI) derived from BAS was the result of ex- 
tensive efforts to systematically evaluate a number of issues addressing the needs 
of a large segment of the affected communities, notably regulatory science. The de- 
velopment of MERSI was the consequence of three previous publications. The first 
formal effort Best Available Science; Its Evolution, Taxonomy and Applications 
(Moghissi et al 2008) contained the fundamental concept of BAS. The next attempt 
led to the publication of the book: Best Available Science: Fundamental Metrics for 
Evaluation of Scientific Claims (Moghissi et al 2010) that in many respect, was the 
second edition of the first book. A new version of that book by Moghissi and 
Swetnam is in preparation. During all of these activities the dominant role of inde- 
pendent peer review in regulatory science was unambiguously described. Con- 
sequently, it was logical to prepare a book Peer Review and Scientific Assessment: 
A Handbook for Funding Organizations, Regulator Agencies and Editors (Moghissi 
et al, in press) with significant applicability to regulatory science. 
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Fundamental Principles of MERSI 

Open-Mindedness Principle: This principle implies that the regulatory science com- 
munity and the general public must be willing to consider new knowledge and new 
scientific claims. 

Skepticism Principle: This principle requires that it is incumbent upon those who 
make a scientific claim to provide sufficient evidence supporting their claim. The 
Skepticism Principle provides balance and ensures that the Open-Mindedness prin- 
ciple is not misused. 

Universal Scientific Principles: The Universal Scientific Principles are a set of basic 
principles and standards that apply to virtually all of the scientific disciplines in- 
cluding regulatory sciences. 

Transparency Principle: Those who make a scientific claim have not only the intel- 
lectual but also the ethical obligation to identify the level of maturity and reliability 
of each segment, and if societal or other areas outside the purview of science are 
included in the claim. 

Reproducibility Principle: Reproducibility is the proof of validity of any scientific 
claim, and separates undisputed areas of science from those that include assump- 
tions and interpretations. 

Pillar: Classification of Scientific Information 

It is well established that science evolves and that new discoveries, advancement 
of scientific knowledge, and numerous technologies result from the evolution of 
science. Therefore, it is necessary to classify scientific information (SI) in terms of 
its level of maturity and its reproducibility. 

Class I: Proven SI: This class consists of scientific laws (or principles) and their ap- 
plication. The scientific foundation of information included in this class is under- 
stood and meets the requirements of the Reproducibility Principle. Scientific laws 
or principles are predictable and reliable. As the majority of SI covered in regulatory 
sciences seldom qualifies as Proven SI, further discussion is not required. 

Class II: Evolving SI. The overwhelming majority of scientific advancements and 
virtually all regulatory science information are included in this class. 

Reproducible Evolving SI: Reliable and reproducible information dealing with a 
subject that is not completely understood constitutes the core of this class. Much 
of medical science provides a good example of Reproducible Evolving Science. Like 
Class I (Proven SI) information in this class meets the Reproducibility Principle. 
However unlike Proven SI, the scientific foundation of information in this class is 
often either unknown or the knowledge is incomplete. 

Partially Reproducible SI: Sometimes referred to as Rationalized SI or Scientific 
Extrapolation this class includes a large segment of regulatory science information 
including predictive models. Although it builds upon Proven or Reproducible Evolv- 
ing SI, it uses assumptions, extrapolations, and default data to derive its results. 
An important characteristic of this class is its level of reproducibility. Whereas the 
scientific foundation of this class meets the Reproducibility Principle the choice of 
assumptions, mathematical processes, default data, and numerous other pre- 
requisites are inherently arbitrary and thus are not necessarily reproducible. 

Correlation-Based SI: This class attempts to correlate systematic observations per- 
formed in accordance with Universal Scientific Principles to an effect. There is an 
extensive literature covering this class including a large segment of epidemiology. 
Experience shows that correlation does not necessarily imply causation and as ex- 
pected, some correlations have correctly identified their cause but others have 
proven to be unrelated. A segment of evidence-based medicine belongs to this class. 

Hypothesized SI: An organized response to an observation, an idea, or any other 
initiating thought process constitutes the core of this class. This class seldom if 
ever has a scientific foundation. Obviously, this class does not comply with the Re- 
producibility Principle. 

SI based on Judgment: In the absence of scientific information, decision makers 
may call upon scientific experts to make an educated judgment. There is an accept- 
ed methodology for this process that involves asking multiple qualified and knowl- 
edgeable individuals to answer specific questions and statistically assessing the re- 
sults. Even so, the results are still tantamount to an educated guess. 
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Speculation: Speculation does not meet the standards for any of the discussed 
classes of scientific information addressed above. It is based solely on the opinion 
and intuition of an individual. Often the objective of speculation is to initiate a re- 
search project or stimulate a scientific discussion. 

Fallacious Information: Most unfortunately, the scientific community and the gen- 
eral public are often provided fallacious information presented as science. Often 
called “junk science” or “pseudo science,” some of the information provided to the 
regulators by special interest groups qualifies as fallacious information. 

Pillar: Reliability of SI 

This Pillar requires a formal and generally acceptable process to categorize the 
reliability of SI. Consequently, SI is divided into several distinct categories in as- 
cending level of reliability 

Category I: Personal Opinions. Expression of views by individuals regardless of 
their training, experience, and social agenda are seldom reliable. 

Category II: Gray Literature. Reports prepared by government agencies, advocacy 
groups, and others that have not been subjected to an independent peer review are 
included in this category. Gray Literature is often no more reliable that personal 
opinion. 

Category III: Peer-Reviewed SI. The acceptability of a scientific claim requires that 
it has been subjected to independent peer review and has passed the strict scrutiny 
by independent scientific peers. Peer review is a well established process and 
issued extensively in scientific publications and grant submission. Briefly, an ac- 
ceptable peer reviewer is an individual who is capable of understanding and per- 
forming the project under review with little or no additional study. Furthermore, 
the reviewer must also be independent and without conflict of interest. Finally, 
(ASME/RSI2002) those who have a stake in the outcome of the review may not act 
as reviewers or participate in the selection of the reviewers. Despite its acknowl- 
edged shortcomings peer review is the only available mechanism to assess the va- 
lidity of a scientific claim, aside from reproducing the actual claim. 

CategoryIV: Consensus-Processed SI. In the consensus process an expert panel, con- 
vened in a manner similar to that described for Review Panels, evaluates the pro- 
posed information. Since much of regulatory science falls into the Rationalized, 
Correlation-Based, or Hypothesized SI, it is not surprising that contradictory infor- 
mation can be found in peer-reviewed literature covering a specific subject. In such 
cases, the consensus process increases the likelihood that its outcome would be 
consistent with the information that will result from relevant future studies. 

Pillar: Outside the Purview of Seienee 

One of the most often violated requirements of regulatory science is the inclusion 
of societal objectives, ideology, beliefs, and numerous other non-scientific issues. On 
occasion, the regulators claim that they must include societal objectives in their sci- 
entific activities to be protective of human health, the ecosystem, and numerous 
other worthwhile goals. What is being overlooked is that all of these goals, as desir- 
able as they might be, are outside the purview of science and must be addressed 
after the scientific issues have been resolved. The confirmation of this Pillar is pro- 
vided by the Ruckelshaus Effect (Ruckelshaus 1983, Moghissi et al in press) which 
states that "... all scientists must make it clear when they are speaking as sci- 
entists -ex cathedra-and when they are recommending policy they believe should 
flow from scientific information ...” 

Ethics of Regulatory Science 

One of the key issues needing the consideration of legislators and regulators is 
compliance with ethical principles of regulatory science. Only these principles were 
only recently formulated, they are readily derivable from ethical principles of vir- 
tually all professions notably scientific, engineering, and medical professions. 

Principle I: 

A scientific issue is settled when anyone with the necessary scientific skills, required 
equipment, and facilities can reproduce it. 

On more than one occasion proponents of an issue claim that “science has spoken” 
or “science is settled” or several other phrases indicating that the scientific part of 
a regulatory process has been clarified. In effect, those who make such a claim must 
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provide evidence that the science is reproducible and in the MERSI system, falls 
into Proven or Reproducible Evolving SI. 

Principle II: 

Those who prepare a regulatory science document must provide to the affected com- 
munity assumptions, judgments, and similar parts in a language understandable to 
a knowledgeable non-specialist. 

This principle includes the consequences of using “assumptions, judgments, and 
similar parts,” the justification of using them, and potential alternatives that were 
not used. This principle is based on the MERSI principle on transparency. The regu- 
lated community, the scientists and their organizations, and the interested members 
of the public are entitled to know the regulatory science is used in a specific deci- 
sion. 

Principle III: 

Regulatory science information must exclude societal objectives thus violation the 
MERSI Pillar “Areas Outside the Purview of Science.” 

During the initial phases of the EPA, the need for rapid promulgation of regulations 
led to “being protective” and included societal judgments in the scientific process. 
One can argue if during that period those actions were justified. However the inclu- 
sion of societal objectives or ant other subject that is included in “areas outside the 
purview of science” is not justified. 

Principle IV: 

Regulatory science information is only then acceptable if it has been subjected to 
independent peer review and the review criteria (questions provided to peer reviewers) 
include compliance with principles I, II, and III of regulatory science ethics. 

There is a consensus within the scientific community that peer review is a pre- 
requisite for acceptability of scientific claims. However, the peer review of regulatory 
science information is particularly important because of the usage of “assumptions, 
judgments, and similar parts.” It is crucial to ensure that the selection of “assump- 
tions, judgments, and similar parts” is not based on a preconceived desire of the reg- 
ulatory science participants to promote a specific goal. Similarly, if societal objec- 
tives are included in regulatory science information, they should be not only identi- 
fied but also justified. 

Proposed Roadmap for Fostering Quality Scienee at the EPA 

Before addressing the proposed roadmap, it is imperative to recognize that the es- 
tablishment of the EPA and actions taken by that agency, resulted in a cleaner and 
healthier environment. It would not be constructive to evaluate the performance of 
the EPA with the objective to see if EPA could have done a better job. Instead, it 
is more productive to propose relevant R&D with the objective to improve EPA’s 
performance by enhancing the transparency of the regulatory science used by that 
agency. 

It is proposed to enact the Regulatory Scieuce Suushiue Act as a segment of 
the EPA authorization/Appropriation or as a separate Act. The proposed Act would 
require that EPA develop processes, procedures, and methods for each regulatory 
decision that is based on or includes science: 

1. Identification of assumption judgments, default data, or other similar systems 
used in the regulatory process, identification potential alternatives, and how 
the conclusion would be different if alternative assumptions, judgments, and 
similar parameters were used. 

2. Description of the content of all mathematical formulations in words. 

3. The information identified above must be written in a language that is under- 
standable to a knowledgeable non specialist or, better yet, to an average per- 
son. 

4. Clear and unambiguous justification for the inclusion of societal objectives in 
science rather than addressing societal objectives in the administrative deci- 
sion process. 

5. Obligation of the EPA to comply with ethical requirements of regulatory 
science 

The Regulatory Science Sunshine Act would require that EPA makes a con- 
certed effort to develop relevant processes, procedures, and methods to respond to 
the needs identified above. As many other regulatory agencies face the same prob- 
lem, such an effort would also benefit numerous other agencies. 



25 


Consequences of Regulatory Science Sunshine Act 

The opposition to transparency in regulatory science is based on the following: 

1. There are those who believe that the “average citizen” is not educated enough 
or smart enough to appreciate the intricacies of regulatory science. 

2. Some of the staff members of regulatory agencies consider that items identi- 
fied under Regulatory Science Sunshine Act to be burdensome. After all, 
whereas scientists in regulatory agencies have a unique competency, others 
do not have relevant experience and competency. 

3. The identification of potential uncertainties would result in the opposition of 
the public to the relevant regulation. It is being claimed that people would 
suggest that in view of these uncertainties no money should be spent to pro- 
mulgate or comply with a specific regulation. 

4. Certain lobbyists with access to regulatory agencies prefer the current situa- 
tion because they can impact the regulations without the remainder of the so- 
ciety having the ability to judge the foundation of decisions without signifi- 
cant efforts. 

5. Members of a variety of advocacy groups also prefer the current situation, as 
long as the political leadership is supporting them. 

6. There are numerous other individuals and groups who are either opposed to 
transparency or do not care one way or another. 

A closer look at the items identified indicates that the following issues are legiti- 
mate and must be addressed: 

Ability of the Public to Follow Regulatory Science: It is true that a segment of popu- 
lation will have difficulties following the intricacies of regulatory science. However, 
other segments are capable of comprehending the subject. In addition using as my 
former boss William Ruckelshaus quoted Thomas Jefferson “If we think [the people 
are] not enlightened enough to exercise their control with a wholesome discretion, 
the remedy is not to take it from them, but to inform their discretion.” 

Competency of Regulatory Agency Staff: There is ample evidence indicating that 
there are scientists outside the regulatory agency who are as competent or more 
competent in relevant areas of regulatory science than the staff members of the rel- 
evant agency. This subject is well recognized by reliance upon peer review. 

Decisions Based on Uncertain Scientific Information: By far the most critical issue 
in the proposed legislation in the legitimate issue of convincing the public that a 
decision is necessary in the interest of the society. It should be recognized that soci- 
etal decisions based on incomplete and uncertain scientific information is more com- 
mon than may appear. 

The example of meteorology can be used to demonstrate the point, a discipline 
that provides short term weather forecasting. Most cities rely upon forecasts on 
snow and its severity and use them to mobilize the necessary personnel and ensure 
availability of relevant equipment. Similarly, governmental agencies make decisions 
on both positive and negative consequences of the predicted rainfall. 

Let us use the example of Hurricane Irene to demonstrate the point. Events re- 
lated to this hurricane started at about August 15, 2011 and a few days later, it 
became clear that Irene would impact the U.S. The pathway of Irene was modified 
as the hurricane moved closer and its severity was modified several times from cat- 
egory I to category II and Category III but as Irene landed it was largely category 
I. Many cities and communities had to make decisions based on the information 
they received at any given time in every case the information was uncertain and 
incomplete until Irene landed. Should the decision makers wait until they had com- 
plete and fully reliable information? No responsible decision maker would do so. 
Conversely, often the predicted weather proves to be wrong. How often a sunny day 
is predicted and how often rain or snow is predicted but the predictions prove to 
be wrong. 

The EPA and other regulatory agencies have the legal and ethical obligation to 
inform the public to the best of their ability the status of the science used in their 
regulatory decisions. The information must include assumptions, judgments, the in- 
clusion of default data, and any other information that impacted the scientific as- 
pects of their decision. 

Conclusions 

The Regulatory Science Sunshine Aet would require a reorientation of the 
EPA’s R&D with the objective to develop processes, procedures, and methods for 
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transparency in regulatory decisions. EPA should be required to identify assump- 
tions, judgments, default data, or other similar systems used in the regulatory proc- 
ess, identify potential alternatives, and how the conclusion would be different if al- 
ternative assumptions, judgments, and similar parameters were used. In addition, 
EPA should attempt to describe the content of all mathematical formulations in 
words. Furthermore, the Act should mandate that EPA makes a concerted effort to 
describe these activities in a language that is understandable to a knowledgeable 
non specialist or, better yet, to an average person. 
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Chairman Harris. Thank you very much. 

I now recognize our third witness, Dr. Kenneth Green, a Resi- 
dent Scholar at the American Enterprise Institute. 

STATEMENT OF DR. KENNETH GREEN, RESIDENT SCHOLAR, 
AMERICAN ENTERPRISE INSTITUTE 

Dr. Green. Thank you. Chairman Harris, Ranking Member Mil- 
ler, Members of the Subcommittee, I thank you for inviting me to 
testify today. I am as the chairman said Kenneth Green, a Resi- 
dent Scholar with the American Enterprise Institute. I am a biolo- 
gist and environmental policy analyst by training, and I have stud- 
ied public policy with a science component for about 20 years now, 
mostly in non-profit and non-partisan public policy research insti- 
tutions across North America. 

My testimony represents only my views and should not be associ- 
ated or construed as the official position of anybody else, and my 
spoken testimony is an abstract from the written testimony I sub- 
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mitted, which is a little bit lengthier, and I wouldn’t have time for 
it today. 

For the sake of complete disclosure, I want to mention that ver- 
bally what I submitted in my truth in testimony form, I have 
served as a grant reviewer for the U.S. EPA on three or four occa- 
sions and was paid their customary per-diem for participating in 
such reviews in 2008, and 2009. Other than offering my judgment 
on the quality and soundness and relevance of potential grant pro- 
posals, I had no other involvement with the award process, nor any 
involvement to my knowledge of any grant recipients or applicants 
in any way. 

In fact, I would like to take a second to compliment EPA on the 
rigor of the process they use to evaluate these outside grant pro- 
posals and innovative grant requests. I was quite impressed with 
the level of rigor and discipline involved in analyzing the proposals, 
ranking them, discussing them, the knowledge ability of the panel- 
ists and so forth, and I commend that process. 

I wish I could say that all of EPA’s science-related activities were 
equally satisfying to think about, but unfortunately, I can’t. My 
own research, both my own research and reading in the literature 
suggests that EPA has serious problems in the way that it employs 
scientific information when it assesses both the potential benefits 
and potential costs of existing and proposed public policies. 

As is common in the public health community, EPA’s science cul- 
ture seems highly risk averse, so much so that when confronted 
with a range of possible risks, they tend to accept assumptions and 
design protocols, analytical protocols and frameworks in ways that 
lead to ever-greater estimations of health risk from ever-lower lev- 
els of exposure to environmental pollutants. This is sometimes re- 
ferred to simply as being conservative or precautionary. In a med- 
ical context that can be beneficial, and indeed, nobody wants the 
agency to blithely dismiss risks, but when such artificially-elevated 
risk estimates are translated into economic estimates of regulatory 
benefit and cost and used to prioritize agency efforts and activities, 
the product is increasingly costly regulations that do increasingly 
little good, or worse in some cases, actually impose costs to society 
greater than the benefits that they produce. 

That is where things diverge from a harmless precautionary ex- 
ercise into poor public policy, and I think it is a serious problem. 
Without a sound understanding of the proposed benefits and costs 
of regulations, it is impossible to have a rational public policy de- 
velopment process. 

It is also difficult for agencies without that to determine their 
regulatory priorities. Thus, even where an agency’s proposals might 
do more harm than good, they can’t optimally bring their resources 
to bear where they can do benefit to secure the biggest bang for 
the regulatory dollar without wasting public revenues, public re- 
sources. 

My own experience with EPA is partly back in 1997, when I was 
looking at the National Ambient Air Quality Standards, and I no- 
ticed some of the problems with the way EPA handles risk assess- 
ment. While EPA on one hand was saying that, well, if we reduce — 
we want to stop the reduction of high-level ozone in the atmosphere 
because that ozone protects people from cataracts and skin cancer. 
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they did not want to account for that same phenomenon when they 
said they want to reduce ozone in lower levels of the atmosphere. 
And yet the reduction of ozone would have the same effect. 

So they didn’t want to net out the benefits and harm, possible 
harms of the regulation, and that was pointed out by others, and 
I think eventually was discussed in the Supreme Court, that you 
have to do a net assessment of health and risks and benefits. 

Others have documented bigger problems. Garrett Vaughn in 
2006 examined EPA’s claims about the benefits and costs of their 
air quality regulations and observed that EPA’s estimate of having 
saved the country some $22 trillion through public health protec- 
tion from 1970 to 1990 would, if accurate, roughly equal the aggre- 
gate net worth of all U.S. households in 1990. Vaughn points out 
that by EPA claims net benefits equal to nearly three times the 
profits of all U.S. corporations and a return on capital by the EPA 
regulations of 500 percent, comparing to a seven percent rate re- 
turn on investment in the private sector. 

Economist Anne Smith recently testified to this Committee, Sub- 
committee, one associated interesting effect of EPA’s benefits infla- 
tion is that the degree to which it makes the total number of 
deaths attributable to particulates implausible. EPA’s presumption 
that fully 320,000 deaths in the United States were due to particu- 
late matter in 2005 represents over 13 percent of all deaths in the 
United States on average. And behind this average is the presump- 
tion that in large expanses of the Eastern U.S., between 16 and 22 
percent of all deaths in 2005 were due to particulate matter. 

The distortions go on, and I could continue, but I am running out 
of time. I am sure we all agree that protecting the environment is 
very important. Having grown up with asthma myself in the San 
Fernando Valley in California, smog capital of North America at 
the time, I am very aware of the role that poor air quality can play 
in determining one’s quality of life. 

But I hope we also agree there is no benefit in overprotection, 
and there is no benefit in misallocation of resources, and there is 
no benefit in the misdirection of attention to small risks at the ex- 
pense of large risks, which we face elsewhere in the economy, and 
that getting the science right is important. 

EPA’s use of science tends to overestimate the risks humans face, 
I believe, and underestimate the cost of compliance and regula- 
tions, leading to poor public policy development. 

With that I look forward to your questions. Thank you very 
much. 

[The prepared statement of Mr. Green follows:] 

Prepared Statement of Dr. Kenneth Green, Resident Scholar, American 
Enterprise Institute 

The views expressed in this testimony are those of the author alone and 
do not necessarily represent those of the American Enterprise Institute. 

Chairman Harris, Ranking Member Miller, Members of the Subcommittee: 

Thank you for inviting me to testify today. I am Kenneth P. Green, a Resident 
Scholar at the American Enterprise Institute. 

I am a biologist and environmental policy analyst by training, and I have studied 
public policies with a science component for nearly 20 years now, mostly at non- 
profit, non-partisan public policy research institutions across North America. I 
began career studying air quality regulations in California, and later expanded that 
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focus to national air quality policy, climate policy, and energy policy, which are inex- 
tricably related. 

My testimony represents my personal views only, and should not be construed as 
the official position any other persons or organizations with which I may be affili- 
ated. 

For the sake of complete disclosure, I want to mention verbally what I submitted 
in my “Truth in Testimony” form: I have served as a grant reviewer for the United 
States Environmental Protection Agency on 3 or 4 occasions, and was paid their cus- 
tomary per-diem for participating in such reviews in 2008 and 2009. Other than of- 
fering my judgment on the quality, soundness, relevance, and potential of grant pro- 
posals, I had no other involvement with the award process, nor, to my knowledge, 
have I had any involvement with any grant recipient or applicant in any way. 

In fact, I would like to take this opportunity to compliment EPA on the rigor of 
the process employed for the reviews in which I participated, which involved assess- 
ing relatively low-cost research proposals by university researchers and small busi- 
nesses in the area of environmental technology, energy technology, and related re- 
search. The level of professionalism I encountered in my review sessions was re- 
freshing. The reviewers selected were clearly knowledgeable, diverse, and applied 
serious effort to the analysis of research proposals that sought teixpayer funding. 
These were satisfying exercises after which I felt confident that, if the EPA followed 
the conclusions of the reviewers, taxpayer money might provide some net social ben- 
efit. 

I wish I could say that all of EPA’s science-related exercises are equally satisfying 
to think about, but unfortunately, I cannot. Both my own research and reading in 
the literature suggests that EPA has serious problems in the way it employs sci- 
entific information when it assesses both the potential benefits, and potential costs 
of existing and proposed public policies. 

As is common in the Public Health community, EPA’s science-culture seems high- 
ly risk-averse, so much so that when confronted with a range of possible risks, they 
tend to accept assumptions and design analytical protocols and frameworks in ways 
that lead to ever-greater estimations of health risk from ever-lower levels of pollu- 
tion exposure. This is sometimes referred to as being “conservative,” or “pre- 
cautionary.” In a medical context, this can be beneficial, and indeed, nobody wants 
an agency to blithely dismiss proclaimed risks to the public health. 

However, when such artificially elevated risk estimates are translated into eco- 
nomic estimates of regulatory benefit and cost, the product is increasingly costly 
regulations that do increasingly little good, or worse, actually imposes costs greater 
than the benefits it produces. 

This is where things diverge from harmless (if excessive) “risk-aversion” into poor 
public policy, and it is, I think, a serious problem: having a sound understanding 
of the proposed benefits and costs of regulation is a pre-requisite for rational public 
policy development. 

Without rigorous benefit-cost estimates, it is impossible for an agency to deter- 
mine regulatory priorities. Thus, even where an agency’s proposals might do more 
harm than good, they cannot optimally bring resources to bear to secure the biggest 
safety return-on-investment for regulatory investments potentially wasting scarce 
public tax resources. This applies between agencies as well. If agency A uses meth- 
odologies that inflate the risk posed by the things they regulate, they may well draw 
public resources away from agency B, which uses more scientifically accurate risk- 
assessment methods. 

As researchers such as Anne E. Smith, Garrett A. Vaughn and others have ob- 
served, the tendency to overstate risk, leading to over-estimates of regulatory bene- 
fits have afflicted what many would consider EPA’s most important mission: ensur- 
ing that air quality is kept at a level that protects the public health with an ade- 
quate margin of safety. 

In my own research looking at the proposed 1997 revisions to the National Ambi- 
ent Air Quality Standards, I noted similar problems. For example, while EPA was 
arguing that preserving ozone levels in the upper atmosphere offered protection 
against cataracts and skin cancer caused by UV exposure, they did not account for 
the fact that ozone anywhere in the atmosphere offers similar protections. Thus, 
they did not consider that lowering ozone levels would increase some risks while de- 
creasing others. 

Others have documented even larger absurdities, and things have not improved 
over time. 

In 2006, Garrett A. Vaughn examined EPA’s claims about the benefits and costs 
of their air quality regulations. 
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Vaughn points out that the EPA’s estimate of having saved the country some $22 
trillion dollars through public health protection from 1970 tol990, “If accurate, that 
sum would equal “roughly the aggregate net worth of all U.S. households in 1990.” 
Vaughn points out that hy EPA’s self-promoting calculations, “In 1990, for instance, 
the EPA claims net benefits equal to nearly three times the profits of all U.S. cor- 
porations.” Given how little EPA claimed its regulations cost, the implication was 
that “EPA’s rate of return on capital exceeded 500%, compared to the private sec- 
tor’s 7 percent.” That is an absurd thought which should have triggered an agency 
“reality check,” but clearly did not. ^ 

As economist Anne E. Smith recently testified to this Subcommittee^: 

• EPA is relying to an extreme degree on coincidental “co-benefits” from PM2.5 
reductions to create the impression of benefit-cost justification for many air reg- 
ulations that are not intended to address PM2.5. 

• In 2009, EPA vastly increased the levels of mortality risks that it attributes to 
PM2.5 simply by starting to assign risks to levels of PM2.5 down to zero expo- 
sure, thus “creating” risks from ambient exposures that are well within the safe 
range established by the PM2.5 NAAQS. 

• This single change nearly quadrupled the pool of purported US deaths due to 
PM2.5 that RIAs can now count as “saved” by minor incremental reductions in 
already-low ambient PM2.5 levels projected under new rules. 

• This additional pool of PM2.5-related mortality consists of the most noncredible 
sort of risk estimate, as it is derived from an assumption that a unit of exposure 
at PM2.5 levels well below any observed in the epidemiological studies poses 
just as much risk as a unit of exposure at the higher PM2.5 levels where asso- 
ciations have been detected. 

• With this change, EPA is now assuming that 13% to 22% of all deaths in the 
Eastern U.S. were due to PM2.5 in 2005, and that 25% of all deaths nationwide 
were due to PM2.5 as recently as 1980. 

• The decision to inflate the PM2.5 risk estimates by presuming risks continue 
down to zero has its greatest impact on co-benefits estimates because — for rules 
that do not address PM2.5 directly — a much greater share of their incremental 
reduction of PM2.5 will occur in areas that are already in attainment with the 
PM2.5 NAAQS (and thus that have PM2.5 levels that EPA has deemed safe). 
Yet, EPA now attributes about 200,000 more PM2.5-related deaths per year to 
exposures in those areas. 

• If it were viewed as credible that such large effects exist below the level of the 
PM2.5 NAAQS, the appropriate policy remedy would be to tighten the PM2.5 
standard, and not to regulate something else altogether in order to obtain those 
benefits through “coincidence.” 

Smith further observed that: 

• “One associated and interesting effect of this benefits inflation, however, is the de- 
gree to which it makes the total number of deaths attributed to PM2.5 implau- 
sible. EPA’s presumption that fully 320,000 deaths in the U.S. were “due to 
PM2.5” in 2005 represents over 13% of all deaths in the U.S. on average. And 
behind that average is the presumption that in large expanses of the Eastern US, 
between 16% and 22% of all deaths in 2005 were “due to PM2.5”. By extension 
(although EPA has not reported this calculation), EPA’s estimates imply that 
about 25% of all deaths nationwide were due to PM2.5 as recently as 1980.” 

I am sure that we all agree that protecting the environment and the health of 
all Americans is an important pursuit. Having grown up with asthma myself. I’m 
keenly aware of the role that poor air quality can play in determining one’s quality 
of life. 

But I hope we also agree that there is no benefit in over-protection, especially 
when such over-protection costs society a great deal of money that could be put to 
better uses elsewhere, such as, in the general economy where it might create jobs, 
which are also important determinants in people’s quality of life. 

EPA’s use of science tends to systematically over-estimate the risks humans face 
from environmental exposures to pollutants such as particulate matter. Combined 


1 Garrett A. Vaughn (2006). “Regulatory Sleight of Hand: How the EPA’s Benefit-Cost Anal- 
yses Promote More Regulation and Burden Manufacturers. (VA: Manufacturers Alliance) 

^Anne E. Smith (2011). “Prepared Statement of Anne E. Smith, Ph.D. at a Hearing 
on”Quality Science for Quality Air” hy the Subcommittee on Energy and the Environment , 
Committee on Science, Space, and Technology, United States House of Representatives, Wash- 
ington, DC, October 4, 2011 
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this with under-estimated compliance and regulatory costs, EPA’s use of science 
leads to inefficient use of scarce public resources, and imposes regulatory burdens 
that may well do more harm than good. To me, this is the core of EPA’s science- 
policy problem, and is probably where any reform efforts should begin. 

I thank you again for this opportunity to testify, and look forward to your ques- 
tions. 

Chairman Harris. Thank you. 

I now recognize our fourth and final witness, Dr. Gary Marchant, 
Professor of Law and Executive Director of the Center for Law, 
Science, and Innovation at Arizona State University. 

STATEMENT OF DR. GARY MARCHANT, 

PROFESSOR OF LAW AND EXECUTIVE DIRECTOR, 
CENTER FOR LAW, SCIENCE AND INNOVATION, 
ARIZONA STATE UNIVERSITY 

Dr. Marchant. Thank you, Mr. Chairman Harris and Ranking 
Member Miller and Members of the Committee. I am delighted to 
be here. My name is Gary Marchant. As was said I am a law pro- 
fessor at ASU, and I direct our Center for Law, Science, and Inno- 
vation, which we bill ourselves as the oldest and largest academic 
center in the country looking at the intersection of law with science 
and technology. And one of the major issues we focus on is to try 
and improve the use of science in various types of legal institutions 
and decision making, including regulatory agencies. I am delighted 
to be here. 

I really want to make two points today. The first one is the im- 
portance of science in our environmental regulatory decision mak- 
ing has never been higher both from the demand and supply side. 
From the demand side that the easy problems in environmental 
policy have been addressed, the things we can smell and see with 
our own eyes and ears and nose. We pretty much have dealt with 
the problems. 

The problems we have left are much more subtle, much more 
long term, and they really require science to tell us the answers. 
We are not going to detect it with our own senses. We can’t tell 
as individuals what is a problem of what we are exposed to. We 
really need science to tell us that. 

So science from a demand side has never been in greater need, 
and then from the supply side we have a lot of new science meth- 
ods coming forward today of toxicogenomics and biomarkers and 
model ecosystem effects and so on. 

So we are getting a lot of new scientific methods coming into our 
environmental policy world as well, and the combination of those 
two I think have made science really critical more than ever before 
in how we deal with environmental policymaking. 

Now, two caveats about that, one, as Ms. Dudley has mentioned, 
science can’t answer the questions. Science is an extremely impor- 
tant input in our environmental decision making, but the decisions 
that EPA has to make are at the end normative. They incorporate 
science but a lot of other factors as well. 

And the second point is that science is always going to have un- 
certainty. You know, there is always going to be things we don’t 
know in science that is inherent to science. And so it is almost as 
important for scientists and science to tell us what we don’t know 
as much as we do know so that the decision makers at EPA and 
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elsewhere can make informed decisions based on that uncertainty. 
That is inherent to their decision making. 

So my second point is that the context and the institutional envi- 
ronment in which science is dealt with really matters. Science has 
these norms of how it works in and of itself, of objectivity, of neu- 
trality, of looking at the data and making our decisions based on 
the data, not on personal values and interests and so on. 

And because scientists are humans we never get that perfectly 
right, but there are different ways in which science is done and dif- 
ferent institutional contexts that make a big difference. If you don’t 
believe me, you know, just go into a courtroom these days and 
watch how science is delivered there. It is completely adversarial, 
and you are not getting the way science is supposed to be done. 
They are getting two hired guns on opposite extremes. 

What you need is a type of process that encourages sort of the 
consensual way that science works best, and when it is done in an 
adversarial or politicized environment, it doesn’t work as good as 
it should. 

And so you get two kinds of problems. You get a blending of 
science and policy, the two overlap, and it is hard to draw the line, 
what Wendy Wagner has called the Science Charade, often where 
agencies will explain a decision as a scientific decision that involves 
a lot of science but also involves some normative decisions that 
should also be expressly recognized. 

And the second one is you get both conscious but even more im- 
portantly subconscious change in your view on the science based on 
the policy preferences of the organization you are in. 

So I think both of those are problems that EPA faces because 
EPA is at root a very political and policy organization. I don’t mean 
that in a derogatory — they have to be that way. They are making 
these complex decisions that involve science but also politics and 
policy and values and all kinds of different factors, and they have 
to work in that sort of messy world. That is not a good world for 
science, and so the idea I would like you to consider is the idea of 
creating some kind of separate institution to deal with the science. 

This was brought up way back in the ’80s, in the so-called Red 
Book, the National Academy of Science considered that, that it is 
important to separate the science and the policy but didn’t agree 
we should put them in separate institutions. 

But what I would like to suggest is that it might be time to reex- 
amine that, to put the science in a separate organization. I have 
written a couple of articles I suggest something called an Institute 
for Scientific Assessments, that would basically do these type of as- 
sessments in a neutral organization that runs on a scientific model 
rather than a more policy-type model. And from the bottom to the 
top, from the DNA of that organization, it works on the scientific 
model. 

And I think there is a couple of really good examples of that we 
can look at. The Health Effects Institute here in the United States, 
Europe created something called the European Eood Safety Author- 
ity that works that same way, and both of those organizations in- 
culcate sort of scientific values top to bottom, and the result is a 
product coming out of them that is respected across the board. 
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And so if we can do that, we can start with this baseline of what 
science can tell us and what it can’t, and then we can go from there 
and make our political and policy decisions, and if we can do that, 
I think we will end up with both good science and good regulation. 
Thank you. 

[The prepared statement of Dr. Merchant follows:] 

Prepared Statement of Dr. Gary Marchant, Professor of Law and Executive 
Director, Center foe Law, Science & Innovation, Arizona State University 

Good afternoon Mr. Chairman and Members of the Committee. I am Gary 
Marchant, a tenured Professor of Law at the Sandra Day O’Connor College of Law 
at Arizona State University. Among other responsibilities, I am the Faculty Director 
of the Center for Law, Science & Innovation at ASU, the nation’s oldest and largest 
academic center studying the intersection of law with science and technology. One 
of the central missions of my Center is to promote better use of scientific evidence 
in legal institutions, including legislatures, regulatory agencies and courts. My testi- 
mony today is therefore closely aligned with that mission, although the views ex- 
pressed here represent my own personal perspective on the question of improving 
science at EPA. 

The Critical Role of Science at EPA 

Science plays a critical role in EPA’s mission to protect human health and the en- 
vironment. Over the past couple decades, the agency’s focus has shifted from the 
“low hanging fruit” of obvious pollution problems that we can all see billowing out 
of pipes and smokestacks to more subtle and uncertain environmental problems that 
we cannot detect with our own senses. Increasingly we have to rely on science to 
inform us about the risks (or lack thereoD from chronic exposure to individual (or 
combinations of) chemicals, low-level exposures to ionizing or electromagnetic radi- 
ation, new materials such as nanotechnology, ecological disruptions, and climate 
change, to name but a handful of the almost unlimited inventory of possible envi- 
ronmental risks. As the demand for scientific inputs into environmental regulatory 
decision-making has grown, so too has the supply of new scientific models, tech- 
niques and methods that could be used in environmental decisions. This trend of 
an increasing demand for, and the supply of, scientific inputs into environmental 
regulatory decision-making will surely continue and even accelerate for the foresee- 
able future. 

While science is critical to EPA’s decision-making, there are two important cave- 
ats about the role of science. First, science alone can rarely if ever decide an envi- 
ronmental issue on its own. While sound science can and should inform the regu- 
latory decision, the ultimate decision on whether, how, and to what level to regulate 
an environmental problem is an inherently normative decision that goes beyond 
science. ^ Thus, agency attempts to justify or defend regulatory decisions as being 
dictated by science is a fallacy that Wendy Wagner and others have described as 
the “science charade.”^ The second caveat is that, without diminishing the role of 
science, the practical reality is that science is always full of uncertainties and gaps. 
Thus, it is almost as important to know what science can’t tell us as it is to know 
what science can tell us. 

The Institutional Context of Science 

While science is critical to EPA’s activities, many have been critical of EPA’s 
treatment of science. Former Deputy Administrator of EPA Robert Sussman wrote: 
“The bottom line is that nobody likes EPA science. Congress does not like it, the 
scientific community does not like it, the environmental groups do not like it, and 
industry certainly does not like it.”^ Even the EPA, in a 1992 assessment of the 
role of science in its own decision-making, concluded that “EPA science is of uneven 


^Cary Coglianese and Gary E. Marchant, Shifting Sands: The Limits of Science in Setting 
Risk Standards, 152 PENN. L. REV. 1255-1360 (2004). 

2 Wendy E. Wagner, The Science Charade in Toxic Risk Regulation, 95 COLUMBIA LAW RE- 
VIEW 1613-1723 (1995). 

3 Robert M. Sussman, Science and EPA Decision-making, 12 JOURNAL OF LAW AND POL- 
ICY 573-587 (2004). 
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quality, and the agency’s policies and regulations are frequently perceived as lacking 
a sound scientific foundation.” * 

The central focus of my testimony is that the institutional context in which EPA 
considers and incorporates science into its regulatory decision-making inevitably 
tilts, and/or is perceived as tilting, its scientific findings in the direction of the agen- 
cy’s political and policy preferences. Science is not supposed to be influenced by such 
policy preferences — that is a recipe for the actual and perceived bias and distortion 
of science. At its ideal, science strives to be as neutral and objective as possible, 
driven by the data itself rather than extrinsic considerations such as politics, policy 
preferences, personal values, and bias.® The closer science approaches that ideal, 
the more useful it is, because it is then informing the decision-maker what we know 
and what we don’t know. Of course, since science is a human undertaking, it never 
achieves its absolute ideal, but my primary comment is that the institutional con- 
text in which science is presented and considered is a key factor for how closely 
science approaches its objective ideal. 

When science is addressed in an advocacy or partisan institutional context, it 
tends to be distorted to fit preferred outcomes, with selective reliance on the data, 
one-sided inferences and assumptions, and uncertainties dismissed or downplayed. 
Science much more closely approaches its objective ideal when it is addressed in an 
institutional context that emphasizes the norms of the scientific community - with 
a preference for consensus-based decisions, an emphasis on the actual data (espe- 
cially if it has been peer reviewed and published in good scientific journals), express 
recognition of the inappropriateness of relying on personal or institutional pref- 
erences or interests, and openly acknowledging uncertainties and limitations of the 
data and resulting findings. 

EPA is an inherently partisan and political organization. This statement is not 
intended to be derogatory or critical. Rather, EPA necessarily and appropriately 
makes decisions that are based on a messy mix of politics, policy, economics, law, 
interests, and values, with a clear and important institutional mission to protect the 
environment and human health. This mixing bowl of facts/policy/values is necessary 
for making ultimate environmental regulatory decisions, but is not a good environ- 
ment in which to develop and evaluate science. Of course, EPA should and does also 
bring science into its decision-making mix, but it would be better if the scientific 
input injected into that decision-making process was developed in a more objective, 
reliable, and credible forum than within the political cauldron itself. In other words, 
it would be best if the science was developed and evaluated separately, and in par- 
ticular in a separate institutional context, from the more political decision-making 
process. 

This issue of whether and how science should be separated from policy and every- 
thing else was addressed in an influential 1983 report by the National Research 
Council (NRC), which is often referred to as the “Red Book.”® That report set forth 
a framework for regulatory risk analysis that has generally been followed ever since 
by U.S. and many foreign regulatory agencies. A central issue in the report was that 
of separating risk assessment, a primarily scientific undertaking, from risk manage- 
ment, a more policy-related undertaking. The Red Book found that “[a]t least some 
of the controversy surrounding regulatory actions has resulted from a blurring of 
the distinctions between risk assessment policy and risk management policy,” and 
accordingly recommended that “regulatory agencies take steps to establish and 
maintain a clear conceptual distinction between assessment of risks and consider- 
ation of risk management alternatives” (p.3). 

While recommending the separation of risk assessment from risk management 
within a regulatory agency, the NRC report recommended against dividing risk as- 
sessment and risk management into separate institutions because of the need for 
risk assessors and risk managers to communicate with each other. While there are 
some benefits to integrating science with policy within an institution, there are also 
clear disadvantages with regard to the objectivity and credibility of science produced 
from such a hybrid organization. As the role of science becomes ever more important 
to era’s mission, and as the perception of EPA’s science continues to be skeptical 
across the political spectrum, it may be time to consider a different model that insti- 


^ Environmental Protection Agency, Safeguarding the Future: Credible Science, Credible Deci- 
sions, Report of the Expert Panel on the Role of Science at EPA. Washington, D.C.: Government 
Printing Office (1992). 

sSee, e.g., ROBERT K. MERTON, THE SOCIOLOGY OF SCIENCE: THEORETICAL AND 
EMPIRICAL INVESTIGATIONS. Chicago: Chicago University Press (1973). 

6 NATIONAL RESEARCH COUNCIL, RISK ASSESSMENT AND RISK MANAGEMENT IN 
THE FEDERAL GOVERNMENT. Washington, D.C.: National Academy Press (1983). 
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tutionally separates the generation and assessment of science from the application 
of that science in regulatory decision-making. 

Successful Examples of Separating Science from Policy 

There are some useful precedents of institutionally separating science from policy- 
making. Two entities that have been successful in this regard are the Health Effects 
Institute and the European Food Safety Authority. 

The Health Effects Institute (HEI) is a nonprofit corporation created in 1980 to 
provide independent research on air pollution issues that is co-funded by EPA and 
the automobile industry. The objective of the HEI was to provide “high-quality, im- 
partial, and relevant science on the health effects of air pollution.”'^ Although HEI 
was initially a purely research organization, it subsequently assumed a secondary 
function of providing neutral scientific assessments of controversial issues. The 
HEI’s commitment to providing a neutral, objective scientific assessment of con- 
troversial air pollution issues, implemented through both its organizational struc- 
ture and procedures, has made it a highly-regarded and credible “honest broker” on 
air pollution science. ® 

The European Food Safety Authority (EFSA) was created by the European Union 
in 2002 to serve as “an independent source of scientific advice and communication 
on risks associated with the food chain.” ^ The structure of EFSA is explicitly based 
on separating science-based risk assessment and policy-based risk management into 
separate institutions: 

In the European food safety system, risk assessment is done independently from 
risk management. As the risk assessor, EFSA produces scientific opinions and ad- 
vice to provide a sound foundation for European policies and legislation and to sup- 
port the European Commission, European Parliament and EU Member States in 
taking effective and timely risk management decisions . . . EFSA’s most critical 
commitment is to provide objective and independent science-based advice and clear 
communication grounded in the most up-to-date scientific information and knowl- 
edge. 

EFSA therefore provides scientific and risk assessments relating to food safety to 
the regulatory bodies of the European Union (i.e., the EU Commission and the EU 
Council of Ministers) as well as individual member nations, and issues such assess- 
ments in response to specific requests or “questions” from its “clients.” While the 
EFSA has not been without some controversy, it has generally been perceived as 
responsible for restoring credibility and public trust to the European regulation of 
food safety after a series of European food controversies. Again, the primary rea- 
son for EFSA’s success is an institutional commitment to scientific objectivity, as 
seen by the commitment in its Mission Statement “to the core standards of scientific 
excellence, openness, transparency, independence and responsiveness.” 

A Proposed Institute for Scientifie Assessments 

To separate institutionally science from policy in environmental regulation deci- 
sion-making, my colleague Angus Macbeth and I proposed in 2008, as part of the 
“Breaking the Logjam” project, the creation of an Institute for Scientific Assess- 
ments (ISA). I have since elaborated on this proposal in an upcoming chapter 
written for a new book to be published in 2012 edited by Professor Jason Johnston 
at the University of Virginia School of Law on the broader topic of improving regu- 
latory science tentatively titled “Institutions and Incentives in Regulatory Science.” 

The ISA would be an independent, stand-alone scientific assessment body that can 
provide highly valuable and credible scientific input into the regulatory process. It 
would be structurally and procedurally designed to limit its activities to scientific 


■'HEALTH EFFECTS INSTITUTE, ABOUT HEI, http://www.healtheffects.org/about.htm (last 
visited Aug. 22, 2008). 

® Terry J. Keating, Lessons from the Recent History of the Health Effects Institute, 26 SCI 
TECH. HUM. VALUES 409-430 (2001).9 EUROPEAN FOOD SAFETY AUTHORITY, ABOUT 
EFSA, http://www.efsa.europa.eu/EFSA/efsa locale-1 178620753812 AboutEfsa.htm . 
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i^See, e.g., Ragnar E. Lofstedt, A European Perspective on the NRC “Red Book,” Risk Assess- 
ment in the Federal Government: Managing the Process, 9 HUM. & ECOLOGICAL RISK AS- 
SESSMENT 1327, 1332 (2003). 
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matters and to resist any temptation to stray into policy advocacy. The ISA would 
be staffed and managed by full-time federal employee scientists hired using an inde- 
pendent process based on scientific merit, and overseen by an external advisory 
board that would include prominent national scientific experts, such as the leaders 
of the National Academy of Sciences (NAS) and the American Association for the 
Advancement of Science (AAAS). 

As Angus Macbeth and I initially described the operation and function of the pro- 
posed ISA: 

This new scientific assessment agency would not conduct its own research, but 
rather would gather, evaluate and assess the existing data in a manner that could 
be used by a regulatory agency in making decisions. The regulatory agencies could 
identify questions on which they needed scientific assessments through an annual 
regulatory agenda, supplemented with ad hoc requests as they arise throughout 
the year (similar to tbe EU Commission’s requests to EFSA or EPA’s occasional 
requests for scientific reviews by the National Research Council or its own Science 
Advisory Board). In addition to requesting risk assessments for specific 
rulemakings, an agency may also request a scientific analysis from the ISA on a 
more general or cross-cutting issue. Congress could also request a scientific opinion 
from the ISA, helping to fill the gap in Congressional science advice since tbe de- 
mise of the Office of Technology Assessment in 1995 (pp. 162-163). 

In conducting its assessment, the ISA would be committed to following the norms 
of scientific inquiry as closely as possible, including objectivity, disclosure of uncer- 
tainties and competing hypotheses, and consensus-seeking. 

As has been tbe experience witb both EFSA and HEI, instilling a culture of sci- 
entific objectivity from top to bottom of the organization will be critical to the ISA’s 
success. In my new chapter about the ISA, I describe the potential benefits of the 
ISA: “First, tbe ISA structure, limiting consideration to scientific data and issue 
only, would squeeze out much of the perceived or actual political and policy influ- 
ence currently afflicting regulatory agency science . . . Second, the ISA approach 
could reduce the “science charade” . . . Because the ISA would provide a credible 
independent assessment about what the science does and cannot tell us, it will be 
mucb harder for regulatory agencies to camouflage their policy preferences as 
science. Thus, regulatory decision-making will be more transparent. A third poten- 
tial benefit of the ISA would be to harmonize scientific assessments of the same 
issue between different federal agencies ...” 

I also acknowledge in my new chapter that the “creation of an ISA would no doubt 
raise a number of administrative and procedural issues. For example, what if a reg- 
ulatory agency wanted to depart from the scientific findings of the ISA? What oppor- 
tunity would there be for public comment and perhaps even judicial review of ISA 
assessments? Could a party challenging in court an agency regulation that relied 
on an ISA assessment raise claims against the ISA assessment on the merits? These 
and other issues would require careful consideration.” However, there are models 
and approaches to address these implementation issues, and the potential benefits 
for improving the credibility of EPA’s science may justify this type of institutional 
change. 

In summary, a proposed Institute for Scientific Assessments, staffed and designed 
to follow the scientific model of objectivity, could enhance the utility and credibility 
of the scientific inputs into EPA’s regulatory decisions. Thank you for considering 
my suggestion, and I will be happy to address any questions you may have. 

Chairman Harris. Thank you very much for your testimony, and 
we will start the round of questions — reminding Members Com- 
mittee rules limit questioning to five minutes. 

The chair will at this point will open the round of questions, and 
I recognize myself for five minutes. 

Dr. Marchant, by the way, it is an excellent point, you know. In 
medicine we have the NIH to do the research. Totally separate, 
about as not politicized as you get, and then, of course, the infor- 
mation is public and can be interpreted whatever way it needs to 
be interpreted. 

Ms. Dudley, you know, one thing that has kind of bothered me 
about some of the science that is presented, because we have had 
science used as justification for policy decisions, and you know, 
phrases such as, well, you would save up to, you know, 300,000 
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cases of asthma a year if only you lowered this particulate matter 
standard or mercury standard or some co-benefit or whatever. 

And as scientist, I mean, if I ever made a presentation at a sci- 
entific meeting and in my results section I said up to something, 
I would be laughed off the stage. You know, they want the mean, 
they want the standard error, the standard error of the mean, 
standard deviation, whatever. 

But you know, with something like that I wasn’t — I have only 
been here a year. I wasn’t here at the last round of reductions in 
pollution, but I will bet you the claim was made that if only we re- 
duced that pollution, asthma will go down by hundreds of thou- 
sands of cases a year, and of course, we all know asthma hasn’t 
gone done. It has gone up. 

So do you think that more of the agency’s science funds should 
go actually toward retrospective research to figure out whether 
EPA regulations in the past, what the exact impact has been and 
whether they have accomplished their environmental and health 
goals? Because I bet you we go look at some claims made ten years 
ago about the incidence of asthma if only we could reduce pollution, 
and we would be very surprised by the lack of result. 

Ms. Dudley. I think that is a very interesting proposal. I think 
one thing that is lacking is the feedback mechanism. I think one 
of the reasons it may be hard to do is that even though there are 
large benefits, and we would all recognize there are large benefits, 
they still may not be observable in the number of asthma cases. 

For example, we have observed tremendous reductions in lead. 
We have reduced lead in the atmosphere, we have reduced it in 
children’s blood, and the reason that we do all that is because there 
is a real link between child blood levels and IQ, and yet even 
though we have done tremendous things, we can’t measure that 
change in IQ even with huge changes in lead levels. 

So it is hard, but I think it is certainly worth doing. I think it 
is a very intriguing idea. 

Chairman Harris. Okay. Dr. Marchant, you had said that, you 
know, that one of the problems is that you would change, you 
might change the view. Say if you blend two things, you change the 
view of science based on policy preference, and I got to tell you we 
have had testimony from the head of research at EPA about, you 
know, outlining, for instance, the need for hydrofracturing research 
to figure out if it is safe. 

Dr. Marchant. Uh-huh. 

Chairman Harris. And I asked them at the last panel, I said, 
look. There have been 1.2 million applications of hydrofracturing. 
Zero incidences of documented drinking water contamination. Now, 
a scientist would say in my — wearing my hat as my old profession 
as a physician, we applied the therapy 1.2 million times, and it was 
good, and it didn’t have an adverse consequence, we would call it 
a miracle drug, and we would sell it everywhere around the world. 

Here the EPA scientist says, no. We have to study the safety of 
it. Is that one incidence where perhaps if you had separated the 
science from the policy preference we might have a cleaner sci- 
entific method or a justification for scientific projects? 

Dr. Marchant. Possibly. I am not an expert on that particular 
issue, but I think there is, just as it is human nature that when 
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you are working for an organization that has a policy mission, 
which EPA does, that your view of the world is affected by that. 
Just like when I was a practicing lawyer, it is affected by what cli- 
ent I worked for. My view of the issue is changed when I start 
working for a client. 

There is just a fundamental matter of human nature, and so I 
might deliberately skew my views to favor my institution, my em- 
ployer, but even more subtle, which I think a lot of good scientists 
will have happen to them, even though they try not to have their 
view intentionally skewed, would be this implicit, this subconscious 
way of how you look at the world affected by your organization you 
are in. 

And so if your organization is one that says, all we want to know 
is what the science tells us, don’t — we don’t care about the policy, 
you might get a different answer than one that very much has an 
important policy. 

Chairman Harris. Yes. Thank you. Dr. Moghissi, you think the 
EPA’s federal, the Scientific Federal Advisory Committee, standing 
committees, ad hoc panels should be recommending policy decisions 
in addition to advising of scientific or technical issues, or should 
there be a firewall between that? 

Dr. Moghissi. I believe it should be a firewall. As you can see 
from my accent I wasn’t born in Alexandria, Virginia, and believe 
me I have — where I come from I recall — I was too young during the 
second World War, but there was a number of exceptionally com- 
petent scientists who where when it came to policy issues or came 
to the societal issues, were frankly stupid. Johannes Stark was an 
Nobel-prize winning, the Stark Effect in physics, one of the most 
important ones, and he was a Nazi. How do you explain that? 

I believe there is a place for science, and there is a place for pol- 
icymakers, and I believe it would be bad idea to mix them. 

Chairman Harris. Okay. Thank you. Let me just ask one last 
quick question. Dr. Green, you know, testimony — I recall testimony 
I think it was one panel or two ago that every dollar spent on regu- 
lation we have $40 in benefit, economic benefit. I just suggested, 
well, it is simple. Let us just go and spend a trillion dollars and 
pay off our debt and then gain some to that. 

Do you think part of the problem is that, again, because the tes- 
timony in front of the Committee in the past has been, well, you 
know, you are going to save up to 300,000 of this or up to 10,000 
of this, and I don’t know what the range is. Maybe the range of 
the real science was you actually might lose lives, you know, to 
save lives. I don’t know. 

Where are they getting these estimates of 30, $40? I mean, how 
do they come up with something — because when I talk to business 
people, they know the real cost of regulation, and they say, no. 
There is a real cost to these regulations. This disparity between 
what the two opposing philosophies believe, that regulation costs a 
lot or regulation saves us a lot. 

Dr. Green. Well, I think you ask an important question, and you 
raise — use an important word, which is estimates. These are esti- 
mates of lives saved, estimates of illness averted, and increasingly 
as others have pointed out, including Anne Smith, who has testi- 
fied for your Subcommittee in the past, EPA bases these estimates 
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on what are called willingness to pay surveys in which they ask 
people, what would you pay to avoid losing a day at work because 
your lungs are too tight? What would you pay to not have a case 
of bronchitis or to not have a case of lung cancer? 

And the problem is this willingness to pay surveys, not only do 
they not make sense when they compare to each other, the evalua- 
tions you get from them are often nonsensical, the literature on 
willingness to pay surveys specifically says that they really are not 
indicators of what people would actually pay in the real world. 

And so you are using a methodology that you are not — ^you know 
is not going to get you what you want but representing it as if 
somebody has actually been challenged to pay. What would you 
pay, and would you pay more to avoid cancer than you would bron- 
chitis? Well, logically you would say yes, but some of these willing- 
ness to pay surveys don’t necessarily bear that out. 

Chairman Harris. Good. Thank you very much. 

And I now recognize Mr. Miller and give you a couple extra min- 
utes here. 

Mr. Miller. Don’t worry about it overly much, Mr. Chairman. 

I am puzzled by the testimony. I think I agree with the funda- 
mental idea that science should not be the only criteria. It should 
inform decisions that policymakers will apply, the information that 
comes from science, the ways in which science informs policy, but 
then there are other criteria, other considerations that come into 
affect. 

But what I did not really get from your testimony is how does 
that then happen? Ms. Dudley said that she didn’t think that the 
agency should make the decision. The EPA has criteria set by Con- 
gress, passed by statute. We are policymakers. Our names actually 
appear on ballets. It is right in the Constitution, Article 1, and it 
sets out how to decide what the criteria are and then apply those — 
apply science to those statutory criteria and reach a result. 

If not the agency, who? And on what basis? Ms. Dudley? 

Ms. Dudley. I didn’t mean to say that agencies shouldn’t make 
a decision. I am not sure what I said that led you to suggest that. 
I certainly think that agencies should make a decision as delegated 
under the constraints delegated from Congress. 

My recommendation was that when Congress delegate authority 
to agencies, don’t do so under the pretense a decision can be made 
solely based on science. Allow the agency to consider, to be able to 
separate the science from a policy decision and base the policy on 
the range of factors that are relevant for making policy. 

Mr. Miller. But those factors should be set out in statute, right, 
and they should be decided by Congress and then applied by the 
agency. Isn’t that the way you think it should proceed? 

Ms. Dudley. By Congress and applied by the Executive Branch. 
Certainly. 

Mr. Miller. Okay. 

Ms. Dudley. And so my recommendation to you was when dele- 
gating to agencies, recognize that there are there are three things 
to think of. One is the pure science. Then there is the risk assess- 
ment that brings together those different assumptions and judg- 
ments, make that transparent and separate that from the policy 
decision. 
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Mr. Miller. Okay. I really did not hear from anyone’s testimony 
how to coordinate science and research at the EPA so that it nei- 
ther creates a bias against regulation or for regulation. How do we 
make sure that it provides useful, detached information that policy- 
makers who wish to apply all the proper criteria can rely upon? 
And I did not really hear that. 

I know that you have an hour-long presentation, Doctor. 

Dr. Moghissi. No, no. My apologies if I wasn’t clear. 

Mr. Miller. Okay. 

Dr. Moghissi. It is the responsibility of the scientists to say that 
is what the science says and the chairman brought it up, he said 
up to so much blah, blah, blah. What do you mean up to so much? 
What is the scientists — we have a statistical process. There is a 
middle point, there is an upper 95 percentile, there is a lower five 
percentiles. The task of the scientist said this is the science. The 
administrator in Ruckelshaus is good at it. You go to the public 
and say me, the administrator, based on the authority provided to 
me by Congress, I am taking this number, and that is my decision. 
The hearing process as you will know is a little more complicated 
than I make it be. In fact, it is, you know, you have to have — an- 
nounce it on 30 days and so on and so forth. 

But he went and said — there was famous hearings in someplace 
in I think in Seattle or in Tacoma, in which he exactly did that. 
This is the science. My conclusion from the science is — that is the 
decision. Like it or not that is my decision. 

Mr. Miller. Any of you really want to address how, in statute, 
we can revise the ERDDAA statute to make sure that we get the 
science that does not have a bias one way or the other? 

Dr. Green, and the bias should also not be anti-regulatory. 

Dr. Green. Well, I agree, and far be it from me to tell you how 
to do you job. I am not a legislator, I never have been. 

Mr. Miller. That is actually what the people at that table do all 
the time. That is your job. 

Dr. Green. I will eschew the activity myself. What I would say 
is there are some problems with transparency that have been men- 
tioned before and transparency and timeliness. One issue I could 
suggest, for example, when EPA does regulatory impact analyses, 
I remember in the 1997, revisions to the National Ambient Air 
Quality Standards, the regulatory impact analysis was not final 
until after the rules were passed and written. 

Therefore, there really was no opportunity for an outside anal- 
ysis of whether or not you would be getting the right return on in- 
vestment. That should not be allowed. I don’t see where you can 
allow a draft regulatory impact assessment to be the only assess- 
ment available before passing something of such far-reaching con- 
sequences. 

So, a suggestion of greater transparency and slowing things 
down. I mean, if you look at the length of the regulations we are 
getting, the complexity levels, as we say, as was mentioned earlier, 
the really easy stuff, getting the photochemical smog out of the air, 
the things you can see. That is done. The things we are working 
on now are much more complicated and yet the regulatory process 
has gotten compressed by timelines that are very tight and make 
it very hard to review and also there are burdens, more obstacles 
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being placed in people who do the reviews. It is not a career-build- 
ing activity for you to participate in reviews for the agency, espe- 
cially when you have to drop everything to scramble to evaluate a 
rule that is put out the evening before Thanksgiving weekend, 
which is all too frequently the case as well. 

Mr. Miller. Okay. Using my last minute of my extended time, 
we will have additional hearings on this subject. We have — the ma- 
jority and minority have been in conversation, have been in discus- 
sions about how to have a balanced panel that will present views 
on how to amend the statute, how to revise the statute that has 
been on the books for 30 years now. 

And who should be at that table to present the proper range of 
use? I mean, now, obviously, for various reasons this panel is of one 
mind, but there are certainly other points of views, and who should 
be at that table to present the proper range of views for future 
hearings? 

Dr. Marchant. I don’t think we are of one mind, but I think that 
it is important to separate out two different issues here, so I am 
not really talking about the research that EPA does. I am talking 
about the science that includes some of their own research but a 
lot of other research and making a regulatory decision. 

And those are sort of two different issues. How they do their re- 
search, what type of research EPA should do itself intramurally is 
one set of issues, which I think is what this statute addresses. The 
issue I address and some of my colleagues here is more how does 
EPA incorporate science from its own laboratories but also outside 
those laboratories in making a regulatory decision? 

Those are two separate issues. You might want to separate them. 

Chairman Harris. Thank you very much. 

The gentleman from California, Mr. Rohrabacher, is recognized. 
You want Mr. Hall? It is your turn but — okay. Mr. Rohrabacher. 

Mr. Rohrabacher. All right. Thank you very much. 

You know, I would just like to start out by reading a quote from 
President Eisenhower in his farewell address, and people always 
look at that address, and I had it — I Googled it up the other night. 
It is 15 minutes long. I just would recommend taking a look at the 
insights of this man who helped defeat the Nazis and saved the 
world from that terrible threat and then came on as leader of our 
country and did in retrospect a pretty good job as President of the 
United States. But he had an incredible vision of things that were 
happening, both before he became President and when he was 
younger and what it might look like in the future. And he warned 
us, yes, against the military industrial complex. Everybody knows 
that. That is a famous part of his speech, but he also spent just 
as much time warning us about the corruption of science. 

And let me read a part of that. “The prospect of domination of 
the Nation’s scholars by federal employment, project allocations, 
and the power of money is ever present and is gravely to be re- 
garded. Yet in holding scientific research and discovery in respect, 
as we should, we should also be alert to the equal and opposite 
danger that public policy could itself become captive of a scientific 
technological elite.” 

And we have lots of evidence of that since Eisenhower left, and 
I just know that we — just in my lifetime I have seen — I remember 
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when we didn’t eat cranberries for Thanksgiving and Christmas be- 
cause some scientist came up with the idea that it was going to 
cause cancer. Of course, it didn’t cause cancer. 

We also, we know that — and these aren’t things that are nec- 
essarily being done intentionally. Maybe this scientist at that mo- 
ment thinks it is really important. We have outlawed DDT, and the 
scientific, all sorts of scientific evidence, and I bet the scientific 
league got behind this because it became very trendy because bird 
shells do, indeed, become less able to protect the developing bird 
with — if DDT is in the environment. But we also know now that 
DDT kills mosquitoes, yeah, and mosquitoes cause malaria, and by 
making sure we protected birds, a number of hundreds of thou- 
sands of birds, we have condemned millions of children in Africa 
to a horrible death of malaria. 

Just examples like this where, you know, you have got scientists 
being used basically to determine policy, and a lot of times their 
policy is wrong. Maybe the science might even be right, but the pol- 
icy is wrong. 

And so I am very sympathetic with Ms. Dudley’s suggestion 
there. I take it from what you are trying to do is trying to separate 
policy decisions from scientific research so the scientists should just 
be answering to the policymakers as to what the exact science is. 
Is that what we are saying here today? 

Ms. Dudley. I think that is part of it, and that has been accept- 
ed since I think it was 1983, when the National Academies of 
Sciences issued a report that said let us separate the risk assess- 
ment from the risk management. So that is part of it, and I think 
some of our statutes don’t allow us to do that. 

But I think then the second part is even in a risk assessment — 
Professor Marchant was talking about this — even in a risk assess- 
ment you have pure science but then you also have some of these 
judgments. And that — you can’t totally separate that, but we 
should at least make transparent where the science ends. 

Mr. Rohrabacher. Well, Malthus was supposedly a scientist. I 
mean, he was the mathematician of his day, and he gave us a chart 
that said that the entire world would starve to death 50 years ago 
if Malthus was correct, and I remember the cyclamates. Remember 
cyclamates? The industry put hundreds of millions of dollars into 
developing a synthetic type of sweetener and somebody said, well, 
it might cause cancer. Then scientists became a trendy thing. They 
outlawed it, by the way, they never outlawed it in Canada. Ten 
years after research they found out it wasn’t cancer causing and 
what ended up, what was the problem then? Well, the fact is high 
fructose corn syrup was developed in-between and put into place, 
and if you want to add — and most of us know, I hope this isn’t 
trendy science as well, but high fructose corn syrup isn’t really 
good for people. My wife has outlawed it in our household anyway. 

So with that said, Mr. Chairman, I want to thank you for making 
sure you had a secondary follow up because it is the EPA decision 
making on science that really can affect so many people’s lives in 
a positive way but also in a negative way. 

Thank you very much. 

Chairman Harris. I thank you. 

I recognize the gentleman from New York, Mr. Tonko. 
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Mr. Tonko. Thank you, Mr. Chair. I thank the panelists for ap- 
pearing before the Committee, and you have all stated your sup- 
port for environmental protection, but you have also offered harsh 
criticism for EPA’s science and EPA’s regulations. 

Could you offer specific illustration of one specific example of 
good regulation and one specific example of bad regulation? 

Ms. Dudley. Do you want me to start? I will start. 

At EPA I would say an example of good regulations was the re- 
moval of lead from gasoline. That was one that was really based 
on clear science and then a clear risk management analysis as 
well. So the science informed the risk assessment, which informed 
the risk management regulatory decision, and I think that has 
brought about dramatic improvements. It is probably the biggest 
success story at EPA. 

I think the Ambient Air Quality Standards are an example of de- 
cisions where we do blur the lines between the science and the pol- 
icy, and that frankly is because the statute says EPA must make 
its decision based on public health and public health alone, and yet 
in the analysis there is no way to draw that line. If you have a lin- 
ear dose response curve that goes to zero, how do you set a stand- 
ard that is requisite to protect public health and choose anything 
other than zero? 

And so that forces, EPA to kind of wave their hands and make 
up all kinds of reasons why they are setting a non-zero standard. 
And so the real reasons for making that decision is obscured. 

Mr. Tonko. Dr. Moghissi. 

Dr. Moghissi. I am glad you asked. The date is 1970-something 
when the Toxic Substance Control Act was passed, and up to that 
time there were no regulations relative to manufacturing of chemi- 
cals, and for all practical purposes the industry could just release 
chemicals as they saw fit. 

The regulations were excellent. They came at the right time and 
resulted in limitations of what the chemical industry could do. And 
they proved to be cost effective, and they were useful. 

Let me give you a bad one. The date is about 1990-something. 
There is a place called Waste Isolation Pilot Plan in New Mexico 
and in which radioactive waste from reprocessing of weapons trans- 
uranic waste is disposed. Now, the way it is set up, transuranic 
waste and chemical waste, hazardous waste were combined. That 
is just the way it worked. And the poor guys had to go and open 
up this highly-radioactive 55 — the container and analyze for chemi- 
cals, the chemicals disposal facility is intended to, I don’t know, 
maybe a couple of hundred years. Waste Isolation Pilot Plan is in- 
tended to — for thousand years. And, yes, they had to do it. And fi- 
nally my organization based on the request of someone who, some 
government agency, we developed way back, we developed a panel, 
which included the guy who wrote hazardous waste regulations, 
and they said, don’t do it. But the EPA had problems and finally 
the only way it happened was the Congress, through appropria- 
tions, said you can’t do it. 

So that was the end of it, and it saved a lot of human life, a lot 
of enormous costs. Can you imagine the requests — requirements for 
opening up the damn, excuse my French, the 55-gallon container 
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and radioactive waste to analyze for a little Benzene or something 
like that? 

Good two examples. 

Mr. Tonko. Dr. Green. 

Dr. Green. Thank you. I think there are two points. I agree with 
Susan, first of all, Professor Dudley, that the handling of lead is 
an example of good regulation. We had an event not that long ago 
in which that was discussed at some length, and I would say there 
is also a distinction that needs to he made between early rounds 
of the National Ambient Air Quality Standards and later rounds, 
in that you are going to have a greater return on your investment 
when you are casing high levels of pollutants, and you have higher 
level effects. 

The wisdom of a regulation can go from being good to being bad 
based on the rounds and evolution of the problem, and a lot of 
times the regulations are not written with end feedback evolution- 
ary mechanisms in mind to prevent that from happening. 

And so I think it is important these things be made. As for 
things that go wrong, it is not just EPA, but you have things where 
it is asbestos regulations which required greater exposures that 
walls had to be torn open that otherwise humans wouldn’t be ex- 
posed to that created risks for the clean-up workers for example. 
That would have to be offset against the risks that people might 
face in exposure to it. 

And finally, approach matters, which is I might agree with you 
that — we might agree on there is a problem. We might even agree 
that there is a regulatory or a need for the government to intervene 
to address that problem, but there is choice options. I can go with 
an eco tax, I could go with a direct emission fee, or I can go with 
a regulation or a technology standard or picking and choosing an 
approach or technology approach. 

And those are going to have different costs and benefit return on 
investment profiles as well as influencing people’s lives and indi- 
vidual liberty and commercial liberty and so forth. 

Dr. Marchant. I think most environmental regulations have 
done more good than harm. My main concern is how they are ex- 
plained to the public and really at the margins, but I think if you 
are looking at beneficial ones, you would have to start with the 
Ambient Air Quality Standards. They have by far the biggest bang 
that we have spent on them, and particularly I think there is some 
issues with the ozone one at the margin, but particulate matter 
and lead and carbon monoxide have all been tremendously bene- 
ficial. The Sewage Treatment Program has been tremendously ben- 
eficial. So there is a lot of examples of good ones. 

There are ones that are bad, and I think it is interesting looking 
at the bad ones to see where it went wrong, but I mean, my favor- 
ite is one, the DC. Circuit struck down a chemical manufacturing 
versus EPA case a few — about a decade ago where EPA modeled 
these high-risk chemicals under the Hazardous Air Program and 
one particular chemical, basically a solid at the modeling character- 
istics that EPA used, and the company, you know, gave them clear 
data that that was a fact, that there would be no exposure, and 
EPA simply ignored it and says, you have got to be regulated like 
this because we are going to assume that at this level, this tern- 
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perature you are not a solid, and they was just clearly scientifically 
wrong, and yet they just went ahead with the regulation anyhow. 

And fortunately, the DC. Circuit struck it down as lacking any 
plausibility in science whatsoever. 

Chairman Harris. Thank you very much. 

The Chairman, Mr. Hall, is recognized. 

Chairman Hall. Thank you very much. 

Ms. Dudley, you are exactly right when you delegated authority 
solely on science. I am sure — I think, I don’t think that anybody 
in here disagrees with that, but I know you didn’t intend to say 
that you ignore good and reliable science, and I respect every one 
of you that are here. You are Professor of Public Policy, Regulatory 
Science, Enterprise, Center for Law, Science, and Innovation. You 
are not with the Administration, and I think Dr. Green did his best 
to compliment the Environmental Protection Agency. I didn’t read 
enough of it. I quit reading it when I gathered that. 

Talk just a minute about the EPA. Maybe — I don’t know how 
long ago, but I was here when we passed the Clean Air Act and 
the Clean Water Act, and we gave the EPA a place in the sun in 
that legislation and had expectations that they would use science 
and use other knowledge that was available to them. And it is my 
opinion that they have deviated from that, and they have deviated 
it to the extent that it is very damaging to institutions in my dis- 
trict and all across this country. 

Dr. Moghissi, in September I sent EPA a letter requesting infor- 
mation on their Integrated Planning Model. You all call it the IPM. 
This is the model that the agency used to analyze the Cross-State 
Air Pollution Rule. You are familiar with that, aren’t you? 

And the assumptions driving this model have never been made 
public, and their knowledge of the model itself has never been peer 
reviewed. 

Now, how would the IPM model fare if evaluated by the ethics 
principle of regulatory science that you described in your testi- 
mony? 

Dr. Moghissi. What they would have to do, they would announce 
the assumptions, every piece of an assumption, and I am familiar 
with half of them but not all of them, many of those assumptions 
on the extreme side of the scientific business, they would have to 
say what would happen if I make a different assumption on the 
other side of it and but how would the conclusion be different. 

That would be subject to independent peer review and if I might 
add, the peer review says you have to be qualified if you are a peer 
reviewer. You do not — may have conflict of interest but those who 
have a stake in the outcome of the peer review may not be peer 
reviewers nor may participate in the selection of the reviewers. 

And in my opinion that is very often violated. Therefore, it is im- 
perative that this Sunshine Act that I am suggesting open up to 
the public. Let them — if the people know what they are doing as 
you are suggesting, if the model, the assumptions in the model are 
described, if the judgments are described, if the — if alternative as- 
sumptions on judgments are described, and the conclusions that 
are from it described, then the role of the science is done. The next 
step is the Administrator has to go and say, I pick up this because 
in my opinion this is the right choice. This is my opinion as an Ad- 
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ministrator of the EPA. That is what I would like to do. It is not 
poor regulation. It is not anti-regulation. It is just the truth in reg- 
ulation. 

Chairman Hall. How would the IPM model fare if evaluated by 
the ethics principle or regulatory science that you described in your 
testimony? How would that be? 

Dr. Moghissi. Thank you, sir. 

Chairman Hall. If you can’t knock that one out of the park, I 
will give you another one in a minute. 

Dr. Moghissi. In other words, the peer review must say, include 
in the peer review not I picked up this assumption, I picked up this 
alternatives, these are why I did it. I did not — if I include societal 
objective in it, that is why I did it. Could I have done without it? 
The openness, the transparency is the key issue. Again, it does not 
change anything in terms of pro regulations or anti regulations. 

What it says is you should tell the public. You gentlemen over 
there are a hell of a lot more qualified on policy than I am, and 
I am a scientist, and I am not necessarily all that modest, but I 
am not qualified. You are qualified. 

So, therefore, in my opinion these ethical requirements and it 
took several years to develop them, are very important to comply 
with. 

Chairman Hall. IPM is the model that the agency used to ana- 
lyze the Cross-State Air Pollution Rule, as you know, and it af- 
fected adversely all over this country but it particularly affected an 
entity in my District at Mount Pleasant, Texas, Luminant case. I 
think you may be familiar with that, but it cost them 500 jobs al- 
most immediately, and it is my opinion and my opinion alone that 
there was such an outcry from that and how it affected a lot of 
other states, not just Luminant, but they began to make some kind 
of a realization that they might have made a mistake in leaving, 
in bypassing or ignoring some science in arriving at their opinions. 

This — we had an EPA assistant administrator, Gina McCarthy, 
here before us, and when she was pressed for some reason, she did 
something there and gave some ruling without her applying the 
proper science to it, but she gave us the arrogant answer, we are 
not in the business of creating jobs, and I left her space to make 
an apology for that in the testimony, but it never came. 

And that is what we get from EPA today. Should EPA continue 
its path to implement the Cross-State Rule given that the IPM 
model is not peer reviewed and it is not transparent, or should 
EPA discontinue their current efforts until such a time that the 
IMP model is peer reviewed and transparent? Is that the way you 
think they ought to go? 

Dr. Moghissi. The way to remedy that, sir, to pass the Regu- 
latory Science Sunshine Act to impose upon the EPA they have to 
be transparent, they have to pass the peer review consistent with 
the requirement of ethics of the regulatory science. 

People are not stupid. The people understand it. The arrogance 
of some of the science within or in policymaking I know better 
what is best for the country. No. The people of the United States 
know what is best for the country, and therefore, the Sunshine Act 
open up to the public, and again, this is not pro or against regula- 
tions. It just opens it up. 
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Chairman Harris. Thank you very much. 

Chairman Hall. My time is up. Thank you, Mr. Chairman. 

Chairman Harris. Thank you very much. 

I recognize the gentleman from California, Mr. McNerney. 

Mr. McNerney. Thank you, Mr. Chairman. Mr. Chairman, I 
would like to address one of the issues you bring up often, hydrau- 
lic fracturing. The EPA was authorized by Congress in 2010 to 
open up a study in hydraulic fracturing and its potential impact on 
groundwater. 

Now, maybe you disagree with that decision, but we saw when 
we were discussing nuclear waste a few weeks before that if the 
public doesn’t buy into a procedure in their district, you are going 
to end up in courts, there is going to be fights, it is going to go no- 
where. And so it was wise, I think, for Congress to authorize the 
EPA because there is a groundswell of public concern about poten- 
tial pollution from hydraulic fracturing. 

So if we can prove that it is not a problem, that is good. If we 
can prove it is a problem, it needs to be done, and so I just wanted 
to address your continued concern about that. 

Now, about the issue that we are discussing today, I didn’t nec- 
essarily disagree with what was said by the panel. I think Ms. 
Dudley’s use of the word, charade, regarding a policy in science 
was needlessly inflammatory. But, one of the things that I kept 
hearing was that we need to have science separated from policy. An 
example we have of that in the 1990s, we have the Office of Tech- 
nology Assessment did unbiased work, and the then Speaker of the 
House of Representative, Newt Gingrich, disbanded that organiza- 
tion because in my opinion he didn’t like what they were doing. 

So separating science and policy is not really possible in this sort 
of political environment. I don’t know exactly where to go from 
there, but the objective of this hearing to decide how we should em- 
power or enable scientific research to inform regulatory decision 
making. 

Now, one of the things that Ms. Dudley says is — and all of you, 
I think, really is that science is not certain, and that is for sure. 
Science is never certain, but there are ways to include probability 
assessments in scientific judgments, and there are statistical tools 
that can be used in decision making with uncertainly. 

I have been — I am a mathematician, so I am aware of those 
tools. So what do you say, Ms. Dudley, in terms of using that ap- 
proach because we can never have absolute certainty in science? 

Ms. Dudley. I think that is an excellent suggestion, and the Na- 
tional Academies have suggested that regularly to improve the 
quantitative uncertainty analysis for regulations that are likely to 
have the biggest impacts. And the reason that is so important is 
then you make more transparent what the assumptions that are 
going into that risk assessment are. 

So I think that is key. 

Mr. McNerney. So we don’t have to have absolute certainty. We 
can’t have absolute certainty. 

Ms. Dudley. Oh, no, you can’t. 

Mr. McNerney. But we can make decisions in the absence of ab- 
solute certainty. 
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Ms. Dudley. Absolutely. I don’t think anyone suggests that you 
have to wait for certainty. In fact, your point is related to the 
science charade, which I did not mean to be inflammatory. It is ac- 
tually a phrase that Wendy Wagner coined a decade or so ago in 
an excellent article. I have been doing some research lately and 
find that all the incentives in the regulatory system are to perpet- 
uate that charade, and the charade is that when we make the pre- 
tenses that something is based solely on science, we are hiding the 
real decisions. 

And what you are suggesting is a way to step back and make 
that clear: here are our assumptions, and here is the range of as- 
sumptions we could have used, and if we do this quantitative un- 
certainty analysis, that makes transparent for everyone. It is an 
excellent suggestion. 

Mr. McNerney. Okay. Mr. Green. 

Dr. Green. I think your point is very well taken. It is also impor- 
tant for people to, I think, to understand. Science is provisional and 
that science has a culture which is rather different than the cul- 
ture of policy, and that is science is provisional in that everyone 
understands that a paper that they put out one week can be over- 
turned the next week. 

Mr. McNerney. Uh-huh. 

Dr. Green. And the consequence is simply that somebody goes 
back to the lab. But once you start entraining the scientific find- 
ings into a regulatory process, it is often hard to reverse the train. 
And so as I was saying before, I think some feedback mechanisms 
and evolutionary mechanisms that allow for the provisionality of 
science to feed back and have faster impacts on slowing the train 
or changing the course of direction would be very important. 

One last comment is I think the transparency issue is very im- 
portant. Proprietary models are a real problem when you have an 
agency that is running propriety health models or having propri- 
etary data that can’t be analyzed by — from outside. It is very hard 
to assess whether or not the work itself really does reflect science 
or reflects simply assumptions that are put into models no one can 
see. 

Mr. McNerney. Okay. Mr. Chairman, I yield back. 

Chairman Harris. Thank you very much. 

Yeah. We will have — we have a little more time left, so we are 
going to have a second round of questions, but it is limited to three 
minutes per Member, and I will have the first questions of the sec- 
ond round. 

Dr. Green, the Federal Advisory Committee Act, which governs 
EPA’s Science Advisory Board and CASAC, the Clean Air Scientific 
Advisory Committee, requires that advisory peer review panels be, 
“fairly balanced in terms of the points of view he represented.” 

However, a recent hearing raised a number of issues related to 
these panels. For example, five of the seven standing members of 
CASAC have recently received EPA funds. They are actually direct 
recipients of funds, and the current CASAC Particulate Matter Re- 
view Panel includes nine scientists who had previously rec- 
ommended lowering the annual standards. So, you know, maybe 
have a little bit of policy preference there. 
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Do you think the requirements for scientific, balanced scientific 
advice is met by these EPA committees at this point in time? 

Dr. Green. As I have seen them, I am not — I guess the answer 
I would say is probably not, but the question is how do you best 
represent balanced points of view, and I think a lot of the times 
the discussion comes down in the wrong framework. That is it is 
industry views versus agency views or university views versus pri- 
vate sector views, public sector versus private sector views. 

Without a recognition that there is expertise on both sides, the 
person doing applied chemistry at Kodak or at a chemical company 
has very detailed, specific information that could inform policy 
process. The question is do we capture those people who have di- 
rect field experience as much as we capture the expertise of people 
with university experience or agency experience or agency culture 
or leaning. 

From that I have seen of review panels I rarely have seen one 
as — you were mentioning the — the Ranking Member was men- 
tioning the lack of balance on the panel here. Imbalanced panels 
seem to be the norm rather than balanced panels. 

Chairman Harris. Thank you very much. Professor Dudley, ear- 
lier this year the U.S. EPA’s long-awaited study of formaldehydes 
toxicity was panned by the National Academy of Sciences panel 
that sharply disagreed with the agency’s conclusions and declared 
the effort in need of, “substantial revision.” The panel stated, and 
I am quoting from their statement. “Overall the committee found 
that EPA’s draft assessment was not prepared in a logically-con- 
sistent fashion, lacks clear links to an underlying conceptual frame- 
work, and does not sufficiently document methods and criteria used 
to identify evidence for selecting and evaluating studies.” 

Now, NAS added that EPA’s chemical assessments have consist- 
ently displayed these problems in recent years. How should the 
EPA modify the Integrated Risk Information System in response to 
the criticisms that were raised by NAS? 

Ms. Dudley. I think the NAS report itself gave some very con- 
structive suggestions, and it comes back to the transparency that 
we have all been talking about, both you and your panel today. 

More transparency so that we understand why decisions are 
made because risk assessments require a lot of those assumptions, 
including which studies do you choose, which models do you choose, 
what default assumptions do you use? If we can make that more 
transparent and perhaps follow Dr. Moghissi’s, those steps to make 
sure that we really do understand the range of assumptions, I 
think that would address the Academy’s concerns. 

Chairman Harris. Thank you very much. 

Mr. Miller. 

Mr. Miller. Ms. Dudley, the IRIS procedures have changed at 
least three times in the last decade. There was sort of an early 
Bush Administration procedure, a late Bush Administration proce- 
dure that I think you were the principle author of, and then the 
Obama Administration has developed their own procedures. 

The criticisms of the formaldehyde decision or the formaldehyde 
decision that was criticized by GAO, under which set of procedures 
was that decision made? 
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Ms. Dudley. First let me say I was not the architect of the pro- 
cedures. It was EPA who changed the IRIS procedures. And OIRA 
was not the architect of that. 

I am not sure. I am sure it was over time. I don’t know that the 
new procedures will change that because the procedures really 
were how you do the vetting, interagency and with the public, and 
that certainly helps. I think bringing, engaging a broader perspec- 
tive will address some of the concerns, but I won’t use all your 
time. 

Mr. Miller. Dr. Marchant, you discussed institutional biases at 
EPA and said that it had sorted science and policies and values 
into one big mess, but there are — there were panels, review panels 
that — advisory panels that had been established by statute, and I 
think maybe by EPA on their own, the scientific, to review sci- 
entific research activities. 

What role can those or do those advisory panels play, those 
boards play? Are they functioning, and how can they or do they 
separate science and policy decisions? 

Dr. Marchant. I think they do help. They are sort of an inter- 
mediate step to have sort of an independent scientific check on 
what the agency is doing to give them advice. I do think they are 
still somewhat beholden to the agency as opposed to a completely 
separate and independent institute that is basically made up of ca- 
reer scientists who work for that institute rather than being ap- 
pointed by an agency. 

I also think that those committees should not be making rec- 
ommendations on policy. I don’t think they should be telling the 
EPA what level to set the standard. They should be telling them 
what the science is and then it is the agency’s job to set the stand- 
ard. The scientist shouldn’t be making those policy decisions. They 
aren’t policy experts. 

Mr. Miller. Okay. Dr. Green, same question to you. 

Dr. Green. Well, I think Dr. Marchant’s point is exactly right, 
which is you do need review agencies. You need review panels, but 
it is difficult not to have them be captured or capture the agency 
that they are associated with if there is some sort of benefit, 
whether it is a prestige benefit or an influence benefit or a poten- 
tial grant benefit or any kind of interaction. It is hard to separate 
those things out. 

It’s not that it can’t be done, but I think that the important thing 
is that they should simply draw the line in saying, here is what 
we read the science as. It is all published, it is all public. Here is 
how we interpret it. Here is what we think the science says is, and 
it is not our job to say what it should be or what should be done, 
and they should hand that — that should be clearly — a clearly polit- 
ical decision by the administrative side of the agencies. 

We did — but instead we have previous decisions that would de- 
fend it as being the decision is dictated or driven by the science. 
It is science based. It is science driven. The science says we must 
X, and science can never tell you what to do. It can only tell you 
what it is. 

And so I guess my, one of my — my only suggestion would be that 
we should ban the phrase, the science says we must, from statutory 
language entirely, and we might get improvement that way. 
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Chairman Harris. Thank you very much. 

The gentleman from California, Mr. Rohrabacher. 

Mr. Rohrabacher. Thank you very much, Mr. Chairman, and let 
me just note, and again, this is a very — this is not a scientific anal- 
ysis of what has been said, it might be impacted by my political 
opinions, but I would suggest that the Office of Technology Assess- 
ment was not eliminated by Newt because it was doing things that 
he didn’t like. No. I was here when that happened. We tried to bal- 
ance the budget, and that was one of the years that we were able 
to balance the budget by eliminating what we thought were extra- 
neous groups that were not doing — the job that they were doing 
was not worth the money that was being paid. 

In the case of the Office of Technology Assessment, they would 
often give us their assessment a year or two after we requested it, 
and it was no longer necessary to hear their opinion, but they 
weren’t doing a good job, and at least that is what those of us who 
voted to eliminate that part of our budget in order to have a bal- 
anced budget. 

I also don’t believe that there was ever a groundswell of public 
concern for fracking, that it might impact on the groundwater. 
There was a groundswell among liberal politicians who controlled 
the House at that time, again, another scientific expertise by politi- 
cians, who wanted to prove that — or who believe in like global 
warming is caused by CO2 and that they would do anything they 
could to prevent more availability of oil and gas because that would 
create CO2, which created global warming. Again, a political deci- 
sion. 

Let me ask my one question, which is at a recent hearing I asked 
Dr. Paul Anastas, I guess you pronounce it. I am sorry if I have 
mispronounced it. The Assistant Administrator for EPA’s Office of 
Research and Development, and the EPA Science Advisor, I might 
add, about new regulations for perchlorate, and this was — basically 
I asked him in the face of decades of scientific work by — from the 
National Academies and other reputable scientific institutions, say- 
ing that no such regulation was needed, why were they moving for- 
ward, and his answer simply, well, it implied that a scientific basis 
was not required for EPA to create regulations. 

I would like to hear from the panel just whether or not — we don’t 
have much time, but a yes or no whether you think that — whether 
or not EPA should require a scientific basis to create a new regula- 
tion. 

Just a yes or no or whatever your quick thoughts are. 

Dr. Moghissi. If they want to develop a regulation, which is 
based on science, that would be absolutely necessary. How else do 
you do? The gentleman over there, he is a physician. How else do 
you go and treat someone with medicine if you don’t know what the 
medicine is? 

So, therefore, the disease and medicine are related to each other, 
and therefore, without having the scientific foundation, the regula- 
tion is arbitrary. 

Dr. Green. I would agree with that. I think you need to have a 
scientific foundation. 

That being said, I mean, there could — you could envision in some 
cases where the science is so clear that something is damaging that 
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setting the level could be based on other factors than that. You 
may not actually have to question the science, but I agree. The key 
thing is if you are going to regulate, you need a rationale for regu- 
lation. If it about environmental protection that suggests you have 
measured something that needs protecting, you determine what 
needs to be done in order to protect it. All of those are scientific 
activities, and without them it is hard to see how you have a ra- 
tionale for your actions as opposed to simply acting randomly. Why 
couldn’t they then just pass any regulation they want to on any 
subject if there is no rationale and no linkage to their mission? 

Mr. Rohrabacher. All right. 

Dr. Marchant. I think personally that is a terrible idea to base 
it not on science, but I am glad that they would explain it. More 
honestly, if there isn’t science to back it up, to be explicit about 
that rather than trying to say that there is science so now we can 
all evaluate it and make our own judgment whether that is a good 
regulation or not, and I would think it wouldn’t be if there is no 
science to back it up. 

Mr. Rohrabacher. Thank you, Mr. Chairman. 

Chairman Harris. Thank you. 

The gentleman from New York, Mr. Tonko. 

Mr. Tonko. I am set. 

Chairman Harris. Thank you. Okay. I want to thank the wit- 
nesses for their valuable testimony and the Members for their 
questions. 

The Members of the Subcommittee may have additional ques- 
tions for the witnesses, and we ask you to respond to those in writ- 
ing as quickly as possible. 

The record will remain open for two weeks for additional com- 
ments from Members. The witnesses are excused. Without objec- 
tion, the hearing is recessed subject to call of the chair to a date 
to be determined for a second day of testimony for this hearing. 

[Whereupon, at 3:40 p.m., the Subcommittee was adjourned.] 
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Answers to Post-Hearing Questions 

Responses by Ms. Susan Dudley, 

Director, Regulatory Studies Center, 
and Research Professor of Public Policy 
& Public Administration, 

The George Washington University 


Questions submitted by Subcommittee Chairman Andy Harris 

Ql. We briefly discussed the potential for retrospective “accountability research” to 
serve as a feedback mechanism about past regulations and their health and en- 
vironmental outcomes. What steps could be taken to ensure that these targeted 
reviews fully and usefully evaluate past regulations to inform future decisions? 

Al. Agencies face little incentive to conduct retrospective evaluations of the accu- 
racy of ex ante predictions of health or environmental effects resulting from regu- 
latory action. Unlike federal programs that don’t get reauthorized if they don’t dem- 
onstrate success, regulations tend to stay in place absent some new action. Congress 
could provide incentives for agencies to examine ex post actuarial data of key out- 
comes (such as children’s IQ, mortality rates, etc.) through sunset provisions that 
linked continued authorization to a demonstration of a regulation’s effectiveness. 
Perhaps a pilot project, where Congress identified a particular regulation or target 
pollutant and health effect for review, would help EPA develop a methodology for 
conducting such reviews. 

Q2. The Bipartisan Policy Center’s 2009 report, “Improving the Use of Science in 
Regulatory Policy” was mentioned several times during the hearing. This report 
made several suggestions that may be useful in guiding this Subcommittee’s ef- 
forts to reform regulatory science, including: 

• “Studies used in the formulation of regulation should be subject to data access 
requirements equivalent to those under the Data Access Act." 

• “The process of conducting literature reviews” and “the process of naming advi- 
sory committees” should be made more transparent. 

• “Agencies should avoid turning repeatedly to the same scientists for service on 
advisory committees.” 

• Executive branch agencies need to “help clarify for both officials and the general 
public which aspects of disputes are truly about scientific results and which con- 
cern policy.” 

• “Policy makers should be wary of conclusions of risk that are expressed as a sin- 
gle number.” 

• “Federal agencies need to experiment with ways to increase the number of sci- 
entists who participate in peer review.” 

• “In presenting the conclusions of literature reviews, agencies and their scientific 
advisory committees need to be as open and precise as possible in discussing lev- 
els of risk and uncertainty.” 

Do you agree with any or all of these recommendations? Do you have any additional 
comments or advice in pursuing these goals? 

A2. Yes, I agree with these BPC recommendations. Congress could support these 
(particularly 4th, 5th and 7th bullets) by explicitly recognizing in authorizing stat- 
utes that good policy decisions depend on a range of information, and avoid dele- 
gating decisions to agencies on the pretense that “science” alone can make the nor- 
mative determination of what policy should be. To quote further from the BPC re- 
port: 

• “The first impulse of those concerned with regulatory policy should not be to claim 
‘the science made me do it’ or to dismiss or discount scientific results, but rather 
to publicly discuss the policies and values that legitimately affect how science gets 
applied in decision making.” (BPC 2009, 4) 

• “Distinguishing between science and policy is not always easy or straightforward, 
and scientists may make choices based on values in the course of their work. 
Nonetheless, policy debate would be clarified and enhanced if a systematic effort 
were made to distinguish between questions that can be resolved through sci- 
entific judgments and those that involve judgments about values and other mat- 
ters of policy when regulatory issues comprise both. This transparency would both 
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help force values debates into the open and could limit spurious claims about, and 
attacks on science.” (BPC 2009, 15) 

Legislators should also take care to limit the role of scientific advisory panels to 
advising on science, and avoid embedding their policy views in their scientific rec- 
ommendations. I would also highlight the following BPC recommendation: 

• “In general, scientific advisory panels should not be asked to recommend specific 
regulatory policies.” (BPC 2009, 5) 

Q3. A recent joint report from the EPA’s Science Advisory Board and Board of Sci- 
entific Councilors recommended that the Agency “include sustainability in its re- 
search vision" in order to allow “EPA to adopt sustainability as a core principle 
to inform decisions and actions.” Is this emphasis on sustainability appropriate 
for EPA’s research and science activities? 

A3. I am not familiar with this report or recommendation. 

Q4. Many of the regulatory activities that EPA is currently undertaking are based 
upon statutes and priorities from several decades ago. In your view, are we fo- 
cusing our attention and scientific resources on the most pressing environmental 
issues? Are there ways that EPA could better prioritize? 

A4. A concerted effort to focus resources on the most pressing environmental issues 
would be a welcome change. In 1987, EPA ranked its regulated activities according 
to the risks they posed to human health and the environment. It found that the ac- 
tivities that commanded the largest share of federal resources and public dollars 
were not the ones that posed the greatest risk. However, the allocation of resources 
tracked public perception of risks very well. (U.S. Environmental Protection Agency. 
Unfinished Business: A Comparative Assessment of Environmental Problems. Feb- 
ruary 1987) To my knowledge, EPA has not repeated this exercise but it would like- 
ly be fruitful. 

Q5. Dr. Marchant recommended the Health Effects Institute or the European Food 
Safety Agency as potential models for conducting independent scientific assess- 
ments as an alternative to the current EPA practices. In your experience, are 
there other governmental or non-governmental organizations that demonstrate 
characteristics in scientific assessment or R&D that could serve as a useful 
model for science reform at EPA? 

A5. I do not believe that an independent group of risk assessors, insulated from 
challenge, would address the problems we see in risk assessment in the U.S. While 
isolating risk assessors might be an appropriate solution to a problem of “politicized 
science,” as the BPC report concluded, “some disputes over the ‘politicization’ of 
science actually arise over differences about policy choices that science can inform, 
but not determine.” EPA’s Office of Research and Development arguably fits the 
Marchant model, yet its IRIS process is widely recognized to be broken, with long 
delays and the perception that policy preferences are embedded in numbers that are 
presented as purely risk-based assessments. 

The peer-review process may offer a better model for reform, where challenge and 
debate are encouraged, leading to more robust hypothesis testing and more reliable 
theories. 

Q6. Many EPA science activities are housed within regulatory offices. For example, 
EPA’s Office of Air and Radiation (rather than the Office of Research and Devel- 
opment) manages the National Fuel and Vehicle Emissions Laboratory, as well 
as the National Air and Radiation Environmental Laboratory. In your view, 
should science activities be organizationally insulated from regulatory activities 
to ensure objectivity and balance? 

A6. The organizational location of the unit doing the analysis is less important than 
requirements for transparency so that all analyses can be examined from different 
perspectives and subjected to challenge. Rather than insulating different p’oups 
from challenge, analyses should be replicable, transparently distinguish scientific in- 
formation from assumptions and policy judgments, and be able to withstand legiti- 
mate challenge. Further, the use to which the scientific assessments are put influ- 
ences the quality of the assessment. Assessments that are factors in policy decisions 
that permit balancing of risks, costs, and benefits (such as conducted under TSCA 
or FIFRA) are more likely to be transparent and objective than those conducted for 
decisions for which they will be the deciding factor (such as NAAQS). 

Q7. Some scientific information that is disseminated by federal agencies is subject 
to specific data quality requirements. Are there additional steps that you think 
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could be taken to ensure that these peer review and data quality guidelines are 
followed or expanded for important scientific information at EPA? 

A7. More conscientious adherence to existing information quality and peer review 
requirements would improve both the information disseminated and the policy deci- 
sions on which that information is based. The most rigorous research I’ve seen on 
agency compliance with information quality guidelines is that of Dr. Richard Belzer 
(a former OIRA analyst). He has found that through FY2010 EPA’s average re- 
sponse times were 166 days for petitions and 316 days for appeals. This record is 
worse than the government as a whole, which has average response times of 148 
days and 186 days, respectively. In its information quality guidelines, EPA com- 
mitted to respond to both petitions and appeals within 90 days. (See his presen- 
tations in November and December 2010 here http://www.rbbelzer.com/presen- 
tations.html) 

Checks and balances from other branches of government (legislative and judicial) 
could provide needed incentives to follow these guidelines. Agencies might be more 
responsive if responding were made a nondiscretionary duty, or if there were pen- 
alties that came into force automatically for failure to meet reasonable response 
deadlines. But it’s also possible that the quality of agency responses would decline 
if responding by the deadline were all that mattered to avoid the penalty. Thus, Dr. 
Belzer argues for incentives that encourage timeliness and quality in agency re- 
sponses, not one at the expense of the other. He suggests compliance might improve 
if agencies could correct errors without having to admit that they were wrong. 

Q8. Nearly all of EPA’s recent Clean Air Act regulations have been justified on the 
basis of two studies that rely on entirely on data from the American Cancer So- 
ciety and the so-called Harvard Six Cities Study. Despite the faet that these data 
sets were developed with government funds and provide the basie Agency jus- 
tification for costly regulations, they are not publicly -available so they can be 
analyzed by other scientists. Do you support making this type of information 
transparent^ In your view, would making these underlying data sets available 
to everyone improve the Agency’s regulatory decisions? 

A8. Yes, as the BPC report stated, “Studies used in the formulation of regulation 
should be subject to data access requirements equivalent to those under the Data 
Access Act.” Risk assessments depend on numerous assumptions and choices, so 
making underl 3 dng data available allows results to be replicated and tested under 
alternative assumptions. It helps decision makers understand the sensitivity of pre- 
dictions to different assumptions, and improves the rigor of the analysis and the 
quality of the regulatory decision. 

Q9. As you know, the Office of Management and Budget (0MB) defines agency sci- 
entific assessments as “highly influential” if they “could have a clear and sub- 
stantial impact on important public policies with a potential effect of more than 
$500 million in any one year.” Assessments that fall into this category are re- 
quired to undergo rigorous and transparent peer-review. However, in many in- 
stances EPA has refused to designate even obvious assessments as highly influ- 
ential — such as its greenhouse gas endangerment finding that is the basis for 
tens of billions in regulatory costs. 

Q9 a. In your experience as head of OIRA, how did you apply this test? Do you think 
the EPA Endangerment Finding eonstitutes a highly influential assessment, 
and if so how can Congress better ensure EPA follows standing 0MB require- 
ments? 

A9 a. The Information Quality Bulletin provides agencies some discretion in defin- 
ing “highly influential,” however EPA’s IG recently opined that the endangerment 
finding should have been designated as such. 

Q9 b. At the Subcommittee’s November 17th hearing, EPA Assistant Administrator 
Paul Anastas said that the ageney has not yet determined whether its study 
on hydraulic fracturing would be designated as highly influential. In your 
view is it advisable for EPA to determine whether a study will be “highly in- 
fluential” and thus subject to greater peer review before it actually begins to 
collect and analyze data? 

A9 b. Yes. As, OMB’s Information Quality Bulletin for Peer Review states with re- 
gard to planning for upcoming disseminations: “The agency shall provide its pre- 
diction regarding whether the dissemination will be “influential scientific informa- 
tion” or a “highly influential scientific assessment,” as the designation can influence 
the type of peer review to be undertaken.” 
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Responses by Dr. Alan Moghissi, 

President, Institute for Regulatory Science 

Questions submitted by Subcommittee Chairman Andy Harris 

I am greatly honored to have been invited to testify before the Subcommittee on 
Energy and Environment of the House Committee on Science, Space, and Tech- 
nology. Your questions and my response to them are as follows: 

Ql. The Bipartisan Policy Center’s 2009 report, “Improving the Use of Science in 
Regulatory Policy” was mentioned several times during the hearing. This report 
made several suggestions that may be useful in guiding this Subcommittee’s ef- 
forts to reform regulatory science, including: 

• “Studies used in the formulation of regulation should be subject to data access 
requirements equivalent to those under the Data Access Act.” 

• “The process of conducting literature reviews” and “the process of naming advi- 
sory committees” should be made more transparent. 

• “Agencies should avoid turning repeatedly to the same scientists for service on 
advisory committees.” 

• Executive branch agencies need to “help clarify for both officials and the general 
public which aspects of disputes are truly about scientific results and which con- 
cern policy.” 

• “Policy makers should be wary of conclusions of risk that are expressed as a sin- 
gle number.” 

• “Federal agencies need to experiment with ways to increase the number of sci- 
entists who participate in peer review.” 

• “In presenting the conclusions of literature reviews, agencies and their scientific 
advisory committees need to be as open and precise as possible in discussing lev- 
els of risk and uncertainty.” 

Do you agree with any or all of these recommendations^ Do you have any additional 
comments or advice in pursuing these goals? 

Al. I am in fundamental agreement with the above items. Before I respond to the 
question permit me to briefly address certain issues that would simplify my re- 
sponse to this and other questions: 

1 . It may be recalled that in my testimony on November 30, 2011, I provided a 
definition for regulatory science which is the generalized version of definition used 
by various organizations. I also provided five principles as the foundation of Metrics 
for Evaluation of Regulatory Science Information (MERSI) and three pillars for sci- 
entific information (SI) derived from the MERSI principles. Furthermore, my testi- 
mony emphasized the need to identify assumptions, judgments, application of de- 
fault data, and other non-reproducible (NR) segments in SI. 

2 . A large number of contested decisions of the ERA are traceable to its history 
and how the regulatory process at the ERA evolved. Rresident Nixon established the 
ERA in December of 1970 by combining a number of organizations from various fed- 
eral agencies. Upon its formation, the ERA faced a number of legally mandated 
deadlines and during the early history of the ERA many laws were enacted or reau- 
thorized including Federal Insecticide, Fungicide, and Rodenticide Act (FIFRA in 
1972), Safe Drinking Water act (SDWA in 1974), Toxic Substances Control Act 
(TSCA in 1976), Clean Water Act (CWA in 1977), and Clean Air Act (CAA in 1977). 
In the overwhelming majority of cases, the scientific information and supporting 
data were insufficient to promulgate regulations based on acceptable science. During 
the Initial Rhase of the ERA’s history that lasted about one decade, the administra- 
tors had no other choice but to use what has become known as Best Available Tech- 
nical Information (BATI). Although in a few cases, they were able to use Rartially 
Reproducible SI, in the majority of cases they were forced to rely upon SI at lower 
level of scientific maturity notably judgment of the ERA staff and ERA consultants. 
In order to be protective of heath and environmental effects of pollutants they chose 
what they called the “conservative” approach and overestimated, and often signifi- 
cantly overestimated the human health end environmental effects of the pollutant. 
During this period, the independent peer review process was virtually unknown. 

3 . The next decade of the ERA’s history could be appropriately called the Explor- 
atory Rhase. That phase started with the reappointment of William Ruckelshaus by 



58 


President Reagan and followed by his successor, Lee Thomas. These administrators 
attempted to move the scientific foundation of regulator decisions from the Initial 
Phase to a process that would be scientifically more acceptable. However, during 
this phase the process used during the Initial Phase continued and the BATI proc- 
ess remained the predominant process at the EPA. 

4. I and many other EPA employees had hoped that the Exploratory Phase would 
quickly come to an end and a final phase would use scientific methods and ap- 
proaches commonly used in most scientific disciplines. Unfortunately, despite an 
enormous level of funding; objections by regulated communities; court cases; and re- 
views and comments by organizations such as National Academies, Bipartisan Pol- 
icy Center, and Institute for Regulatory Science, EPA has yet to leave the Explor- 
atory Phase and often relies upon BATI that was common during its Initial Phase. 

In the following I will try to respond to the items identified in the first question: 

Question: “Studies used in the formulation of regulation should be subject to data 
access requirements equivalent to those under the Data Access Act.” 

Response: The Data Access Act requires that agencies provide information to 
those who request it using the Freedom of Information Act (FOIA). Experience 
shows repeated abuse of FOIA and thus a more appropriate approach would be com- 
pliance with Transparency Principle of MERSI. Congress should mandate that all 
scientific data with the exception of those that would violate laws dealing privacy 
or national security, should he placed on the Internet so that the entire scientific 
community could have access to the information and use the data for scientific as- 
sessments. 

Question: “Agencies should avoid turning repeatedly to the same scientists for 
service on advisory committees.” 

Response: As will he described in the response to the question on peer review, it 
is imperative to ensure that information included in NR/SI is not based on the philo- 
sophical, ideological, or other views of an individual. Therefore, it is necessary to 
avoid seeking repeatedly the services of the same individuals as members of the ad- 
visory committees. 

Question: Executive branch agencies need to “help clarify for both officials and the 
general public which aspects of disputes are truly about scientific results and which 
concern policy.” 

Response: There is a similarity between the views expressed by the report of the 
Bipartisan Policy Center of 2009. It may be recalled that the third pillar of MERSI 
is “areas outside the purview of science.” Mixing science with policy and other non- 
scientific issues is not only undesirable but may have adverse consequences. As 
sated above, it is imperative to separate science from its policy implications. Where- 
as properly performed independent peer review can evaluate the validity of scientific 
claims, and the reasonableness of selection of NR/SI including potential alternatives, 
a scientific panel is not necessarily qualified to judge the acceptability of chosen 
policies or identify potential alternatives. 

Question: “Policy makers should be wary of conclusions of risk that are expressed 
as a single number.” 

Response: This question addresses one of the most important issues that led to 
my recommendation of passing the Regulatory Science Sunshine Act. During my 
tenure at the EPA I managed a risk research program which led to the development 
of the methodology of ecological risk assessment, a process currently used at the 
EPA but also globally accepted. Risk assessment for human health and ecology con- 
sists of several steps including: source assessment; transport and transformation of 
pollutant leading to exposure assessment; exposure-effect assessment also known as 
dose-response function; effect assessment; and finally risk characterization. In most 
cases, the EPA uses the steps following source assessment to regulate emission of 
pollutants. In every step there are assumption, judgments and numerous other 
areas that I called NR in the above description. In virtually every step statistical 
methods can be and often are used. Instead of statistically evaluating the combined 
uncertainties in various steps, EPA often uses the upper 95th percentile in each 
step and then combines the values and statistically evaluates the values. In the 
final step the upper 95th percentile is reported. EPA has all the right to use the 
upper 95th percentile provided the midpoint and lower 5th percentile are also pro- 
vided. 
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Question: “Federal agencies need to experiment with ways to increase the number 
of scientists who participate in peer review.” 

Response: One of the fundamental principles governing peer review and scientific 
assessment is to avoid using an individual repeatedly in these activities. Increas- 
ingly, a rule of thumb has heen established requiring that an individual who is 
qualified to be a “peer” may not be used more than three times .The consequence 
of violating this rule is more significant that it may appear. It may be recalled that 
scientific foundation of EPA’s actions is at best Partially Reproducible SI, and often 
lower level of scientific maturity going down to SI based on Judgment. In all of 
these SI classes, there are assumptions, judgments, application of default data and 
on occasion societal objectives, identified described above as NR/SI. The repeated 
use of an individual institutionalizes his/her selection of NR/SI and societal objec- 
tives. 

Question: “On presenting the conclusions of literature reviews, agencies and their 
scientific advisory committees need to be as open and precise as possible in dis- 
cussing levels of risk and uncertainty.” 

Response: In the past and to some extent today, the EPA reviews the existing lit- 
erature and chooses one or a select number of information as the foundation of the 
scientific decisions. It is necessary for the EPA to include all information, identify 
NR/SI and justify the choices. 

Q2. A recent joint report from the EPA’s Science Advisory Board and Board of Sci- 
entific Councilors recommended that the Agency “include sustainability in its re- 
search vision” in order to allow “EPA to cidopt sustainability as a core principle 
to inform decisions and actions.” Is this emphasis on sustainability appropriate 
for EPA’s research and science activities? 

A2. I remember fondly the establishment of the Science Advisory Board. The task 
of the Board was to provide scientific advice to the EPA managers and review their 
scientific activities. Although I am unfamiliar with the Board of Scientific 
Councilors, I presume that their mission is to provide scientific advice. Sustain- 
ability is a societal goal much like many other societal goals such as energy inde- 
pendency or global food supply that include environmental considerations. The two 
scientific organizations would be well advised to consider the statement by William 
Ruckelshaus who stated “ . . . all scientists must make it clear when they are speak- 
ing as scientists — ex cathedra — and when they are recommending policy they be- 
lieve should flow from scientific information.. What we need to hear more of from 
scientists is science.” Much like many other societal goals sustainability is a highly 
desirable but should be mandated by Congress and not by scientific groups. 

Q3. Many of the regulatory activities that EPA is currently undertaking are based 
upon statues and priorities from several decades ago. In your view, are we focus- 
ing our attention and scientific resources on the most pressing environmental 
issues? Are there ways that EPA could better prioritize? 

A3. I believe that currently, the EPA is focusing its efforts including significant re- 
sources to activities that have marginal human health and environmental benefits 
with high-levels of expenditure. By exaggerating the benefits and underestimating 
the costs EPA tries to justify its decisions. Traditionally, the EPA has attempted to 
address chemical and radiological agents and has left decisions on microbiological 
agents to other agencies. EPA would be well advised to address the exceedingly com- 
plex microbiological pollution. 

Q4. Dr. Marchant recommended the Health Effects Institute or the European Food 
Safety Agency as potential models for conducting independent scientific assess- 
ments as an alternative to the current EPA practices. In your experience, are 
there other governmental or non-governmental organizations that demonstrate 
characteristics in scientific assessment or R&D that could serve as a useful 
model for science reform at EPA? 

A4. There is a distinction to be made between performing scientific assessments 
and having the resulting activity independently peer reviewed. EPA and its contrac- 
tors are capable of preparing scientific assessments. What is missing is a properly 
performed independent peer review of the so prepared scientific assessment. The In- 
stitute for Regulatory Science has performed over 300 peer reviews for government 
agencies at federal, state, and local levels and for Congress. In most cases, the over- 
sight of these reviews was performed by a committee established by the American 
Society of Mechanical Engineers in cooperation with several other professions soci- 
eties. The critical part of thee reviews was the proper formulation of review criteria 
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(questions provided to the reviewers).The subject is too complex to be addressed in 
this response and our upcoming text book on peer review (as references in my testi- 
mony) provides details of the peer review requirements. 

Q5. Many EPA science activities are housed within regulatory offices. For example, 
EPA’s Office of Air and Radiation (rather than the Office of Research and Devel- 
opment) manages the National Fuel and Vehicle Emissions Laboratory, as well 
as the National Air and Radiation Environmental Laboratory. In your view, 
should science activities be organizationally insulated from regulatory activities 
to ensure objectivity and balance? 

A5. As initially designed the sole objective of having scientists and engineers as- 
signed to program offices was to ensure scientific competency within those offices 
so that scientific issues could be addressed within those offices in promulgation or 
enforcing regulations. In order for these Individuals to be informed on the technical 
development, they were permitted to participate in technical activities of their re- 
spective professions. If the above-mentioned laboratories or any other laboratory in 
the EPA is predominantly performing R&D, it should be assigned to the Office of 
Research and Development. 

Q6. Some scientific information that is disseminated by federal agencies is subject 
to specific data quality requirements. Are there additional steps that you think 
could be taken to ensure that these peer review and data quality guidelines are 
followed or expanded for important scientific information at EPA? 

A6. As stated several times both in my testimony and in response to question in 
this document, the scientific foundation of EPA’s decision often include assumption, 
judgments, default data and occasionally societal objectives. Consequently, the re- 
view criteria (questions provided to the reviewers) should include the following: 

• Are the assumptions (defined as information that cannot be reproduced by any 
individual with sufficient knowledge and equipment, if required) justified? 

• How would the conclusions be different if the assumptions chosen by the EPA 
would be replaced by other assumptions? 

• Are the judgments (as defined under assumptions) justified and would the con- 
clusions be different if the judgment chosen by the EPA would be replaced by 
other? 

• Are the chosen defaults data justified and would the conclusions be different 
if the default data chosen by the EPA would be replaced by others? 

• Has the EPA provided justification for including non-scientific criteria (often 
justified for being protective) in the science instead of including it in the admin- 
istrative and policy decisions? 

Q7. Nearly all of EPA’s recent Clean Air Act Regulations have been justified on the 
basis of two studies that rely on entirely on data from the American Cancer So- 
ciety and the so-called Harvard Six Cities Study. Despite the fact that these data 
sets were developed with government funds and provide the basic Agency jus- 
tification for costly regulations, they are not publicly-available so that they can 
be analyzed by other scientists. Do you support making this type of information 
transparent? In your view, would making these underlying data sets available 
to everyone improve the Agency’s regulatory decisions? 

A7. In the interest of transparency, let me start be declaring that the late Ben Fer- 
ris, one of the leaders of the Harvard Six City Study, was a friend and I was briefly 
the project officer for that study. Both studies can be classified as Correlation-Based 
SI and it is highly likely that other investigators using the same data would come 
to different conclusions than those reached by the respective authors. I strongly rec- 
ommend that the raw data from both the Harvard and American Cancer Society 
studies be placed on the web. There is no justification for not making the raw data 
available to other scientists. 
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Responses by Dr. Kenneth P. Green 
Resident Scholar, 

American Enterprise Institute 

Questions submitted by Subcommittee Chairman Andy Harris 

The views expressed in this testimony are those of the author alone and do not nec- 
essarily represent those of the American Enterprise Institute. 

Ql. The Bipartisan Policy Center’s 2009 report, “Improving the Use of Science in 
Regulatory Policy” was mentioned several times during the hearing. This report 
made several suggestions that may be useful in guiding this Subcommittee’s ef- 
forts to reform regulatory science, including: 

• “Studies used in the formulation of regulation should be subject to data access 
requirements equivalent to those under the Data Access Act." 

• “The process of conducting literature reviews” and “the process of naming advi- 
sory committees” should be made more transparent. 

• “Agencies should avoid turning repeatedly to the same scientists for service on 
advisory committees.” 

• Executive branch agencies need to “help clarify for both officials and the general 
public which aspects of disputes are truly about scientific results and which con- 
cern policy.” 

• “Policy makers should be wary of conclusions of risk that are expressed as a sin- 
gle number.” 

• “Federal agencies need to experiment with ways to increase the number of sci- 
entists who participate in peer review.” 

• “In presenting the conclusions of literature reviews, agencies and their scientific 
advisory committees need to be as open and precise as possible in discussing lev- 
els of risk and uncertainty.” 

Do you agree with any or all of these recommendations^ Do you have any additional 
comments or advice in pursuing these goals? 

Al. The suggestions of the Bipartisan Policy Center (BPC) referenced in the Sub- 
committee’s follow-up questions have considerable merit. More transparency; more 
data-sharing; greater diversity of agency reviewers; greater distinctions between sci- 
entific findings and value-driven decisions; more accurate descriptions of proposed 
risks; and greater explanations of uncertainty levels could only lead to better public 
policy development. 

I am not convinced that the BPC’s recommendation to “avoid turning repeatedly to 
the same scientists for service on advisory committees” is either necessary, nor par- 
ticularly feasible. There are, after all, a limited pool of scientists who will have par- 
ticular expertise, time, willingness, and capability of giving quality service on a re- 
view committee. It seems reasonable that an agency might repeatedly turn to a par- 
ticular reviewer who has shown a willingness and ability to participate in previous 
reviews. Should sufficient safeguards be in place to guarantee balanced points of 
view, avoid conflicts of interest, etc., agencies should, I think, have the discretion 
to use a person as a “regular” reviewer. 

Q2. A recent joint report from the EPA’s Science Advisory Board and Board of Sci- 
entific Councilors recommended that the Agency “include sustainability in its re- 
search vision” in order to allow “EPA to adopt sustainability as a core principle 
to inform decisions and actions.” Is this emphasis on sustainability appropriate 
for EPA’s research and scienceactivities? 

A2. EPA’s efforts to insert a “sustainability” agenda into their mission is, I believe, 
politically driven and unwise. I most certainly do not believe that the EPA should 
“include sustainability in its research vision.” “Sustainability” is a highly subjective 
term in both spatial and temporal domains. One can define actions as sustainable 
in a region, or with regard to the entire planet. One can define actions as sustain- 
able for a year, a decade, a century, or for eternity. Such a plastic term would grant 
EPA extremely wide latitude in its ability to regulate activities that are far outside 
what I feel is its only legitimate function, which is to protect human health and 
property from damage via environmental contamination. Additionally, arguments 
about “sustainability,” are often used to promote favored technologies at the expense 
of disfavored technologies, and plays into the hands of both rent-seekers and activist 
governments. Thus, by declaring fossil fuels “unsustainable,” and wind or solar 
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power “sustainable,” EPA could slant the playing field against fossil fuel develop- 
ment (as they already do, to the limits of their abilities). 

Q3. Last year you wrote about one specific example of EPA’s National Center for En- 
vironmental Research providing $1.4 million dollars to recruit people to “dredge 
through EPA’s databases in order to gin up new things for the agency to worry 
about and possibly regulate.” Can you discuss the role of these epidemiological 
associations in establishing major regulations, and whether EPA-funded re- 
search is being overly protective^ 

A3. In any large set of data, one can search for, and usually find, any number of 
correlations. In fact, in very large sets of data, such correlations are virtually certain 
to exist. But correlations do not equal causality, and it’s highly likely that research- 
ers dredging through very large data sets will find statistically significant correla- 
tions that are not causally related. As the saying goes, ice cream sales correlate with 
heat-stroke incidence, but that’s not because ice cream causes heat stroke, it’s be- 
cause they both correlate with hot weather. 

As I wrote in the article you referenced, 

“It is one thing for scientists to identify sick populations, and to investigate what 
it is that might be making them sick. It is another thing entirely to sift through 
large data bases in order to come up with correlations that may have no causal rela- 
tionship. 

And, ever helpful, EPA gives some examples of what such data-dredging exercises 
might look like: 

For example, while air pollution associations with respiratory and cardiovascular 
disease have been studied most extensively, evidence is beginning to emerge of pos- 
sible air pollution impacts on additional health conditions including diabetes, neuro- 
logical disorders, and reproductive and developmental outcomes. Studies also might 
evaluate factors that confer increased sensitivity to air pollution effects such as com- 
promised health status, genetic variants, social and neighborhood conditions, higher 
exposure and others. In addition, some research groups have developed innovative 
methods and models to characterize exposure that might be applied to health effects 
analyses in other cohorts to understand whether certain sources or atmospheric 
components contribute to observed geographic heterogeneity in health-exposure as- 
sociations. 

Further, EPA has specific outcomes in mind. This is not random data dredging, 
which would be bad enough. This program seeks to fund directional data dredging 
that looks only for relationships suggesting that exposures to various air pollutants 
causes harm to human health. In EPA’s words: 

EPA is interested in research to explain heterogeneity in health responses to air 
pollutants. Heterogeneity might be explained by: 1) Individual characteristics and 
other environmental/social conditions that increase the likelihood of an adverse 
health outcome among a subset of the population, [emphasis mine] 

To pay for this innovative regulatory fishing expedition, EPA proposes to give 
away $1.4 million dollars in portions up to $300,000, for projects that could last up 
to three years. 

Now, there’s nothing wrong with trying to ensure that people’s health is protected 
from dangerous air pollutants (in fact. I’d argue that it’s a very legitimate function 
of government), but there is something wrong with organizing taxpayer funded fish- 
ing expeditions to probe for new regulatory potential by seeking out obscure rela- 
tionships in large databases. And those problems are intrinsic to data dredging, an 
frequently abused form of data mining. 

Data dredging, according to Wikipedia, is “the inappropriate (sometimes delib- 
erately so) use of data mining to uncover misleading relationships in data. These 
relationships may be valid within the test set but have no statistical significance 
in the wider population.” Wikipedia gives a particularly relevant example: “Suppose 
that observers note that a particular town appears to be a cancer cluster, but lack 
a firm hypothesis of why this is so. However, they have access to a large amount 
of demographic data about the town and surrounding area, containing measure- 
ments for the area of hundreds or thousands of different variables, mostly 
uncorrelated. Even if all these variables are independent of the cancer incidence 
rate, it is highly likely that at least one variable will be significantly correlated with 
the cancer rate across the area.” 

Or, as the Congressional Research Office explains (in the context of fishing for ter- 
rorists in air-travel databases): 
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Although data mining can help reveal patterns and relationships, it does not tell 
the user the value or significance of these patterns. These types of determinations 
must be made by the user. Similarly, the validity of the patterns discovered is de- 
pendent on how they compare to “real world” circumstances. For example, to assess 
the validity of a data mining application designed to identify potential terrorist sus- 
pects in a large pool of individuals, the user may test the model using data that 
includes information about known terrorists. However, while possibly re-affirming 
a particular profile, it does not necessarily mean that the application will identify 
a suspect whose behavior significantly deviates from the original model. 

Another limitation of data mining is that while it can identify connections be- 
tween behaviors and/or variables, it does not necessarily identify a causal relation- 
ship. For example, an application may identify that a pattern of behavior, such as 
the propensity to purchase airline tickets just shortly before the flight is scheduled 
to depart, is related to characteristics such as income, level of education, and Inter- 
net use. However, that does not necessarily indicate that the ticket purchasing be- 
havior is caused by one or more of these variables. In fact, the individual’s behavior 
could be affected by some additional variable(s) such as occupation (the need to 
make trips on short notice), family status (a sick relative needing care), or a hobby 
(taking advantage of last minute discounts to visit new destinations). 

In other words, with data dredging, it really is a situation of “Seek and ye shall 
find.” It is one thing for scientists to identify sick populations, and to investigate 
what it is that might be making them sick. It is another thing entirely to sift 
through large data bases in order to come up with correlations that may have no 
causal relationship, but that might, nonetheless, cause EPA to spend scarce tax- 
payer money researching the potential linkage, or worse, to endlessly dredge 
through databases in search of ever lower, ever more obscure health impacts to jus- 
tify expanded regulation and EPA intrusion into the economy. 

Q4. Many of the regulatory activities that EPA is eurrently undertaking are based 
upon statutes and priorities from several deeades ago. In your view, are we fo- 
cusing our attention and scientific resources on the most pressing environmental 
issues^ Are there ways that EPA eould better prioritized 

A4. Partly because of regulations, and partly because of the normal march of tech- 
nology, which leads to greater efficiency and environmental cleanliness, most indica- 
tors of environmental quality have improved dramatically in the last 40 years. The 
lowest hanging fruit of environmental protection have been plucked, the largest 
risks and degradations mitigated. We now chase after ever smaller risk-reductions, 
and the EPA seems to think that no pollution of any kind is acceptable anywhere, 
for even transient moments where no persons are present. A zero-risk and zero-con- 
tamination mindset has us spending ever greater sums for ever smaller benefits. 
Contrary to EPA’s self-congratulatory analysis, economists widely recognize that 
regulation imposes a drag on economic growth. It is never a good idea to waste pub- 
lic funds, but it is unconscionable to do so under the kind of economic conditions 
we face today. Policy analysts point out that regulations also impose burdens on 
people and by restricting their liberty, can deprive them of opportunity. 

Thus, I think it is very important, as your question mentions, that EPA’s activities 
should focus on areas of greatest environmental risk — and even better, focus on ap- 
proaches to reducing those highest-risks with as little economic destruction as pos- 
sible. 

Unfortunately, as I mentioned in my testimony, EPA’s questionable use of science — 
particularly risk assessment as it relates to particulate matter and low-dose expo- 
sures to various airborne toxics — drives the agency’s prioritization and activities. 

To quote from my testimony: 

As is common in the Public Health community, EPA’s science-culture seems high- 
ly risk-averse, so much so that when confronted with a range of possible risks, they 
tend to accept assumptions and design analytical protocols and frameworks in ways 
that lead to ever-greater estimations of health risk from ever-lower levels of pollu- 
tion exposure. This is sometimes referred to as being “conservative,” or “pre- 
cautionary.” In a medical context, this can be beneficial, and indeed, nobody wants 
an agency to blithely dismiss proclaimed risks to the public health. 

However, when such artificially elevated risk estimates are translated into eco- 
nomic estimates of regulatory benefit and cost, the product is increasingly costly 
regulations that do increasingly little good, or worse, actually imposes costs greater 
than the benefits it produces. 
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This is where things diverge from harmless (if excessive) “risk-aversion” into poor 
public policy, and it is, I think, a serious problem: having a sound understanding 
of the proposed benefits and costs of regulation is a prerequisite for rational public 
policy development. 

Without rigorous benefit-cost estimates, it is impossible for an agency to deter- 
mine regulatory priorities. Thus, even where an agency’s proposals might do more 
harm than good, they cannot optimally bring resources to bear to secure the biggest 
safety return-on-investment for regulatory investments potentially wasting scarce 
public tax resources. This applies between agencies as well. If agency A uses meth- 
odologies that inflate the risk posed by the things they regulate, they may well draw 
public resources away from agency B, which uses more scientifically accurate risk- 
assessment methods. 

Q5. Dr. Marchant recommended the Health Effects Institute or the European Food 
Safety Agency as potential models for conducting independent scientific assess- 
ments as an alternative to the current EPA practices. In your experience, are 
there other governmental or nongovernmental organizations that demonstrate 
characteristics in scientific assessment or R&D that could serve as a useful 
model for science reform at EPA? 

A5. I have not made a formal study of how agencies other than EPA manage risk 
assessment, so I will yield to Dr. Marchant on this matter. I believe that other agen- 
cies, such as OSHA take a more pragmatic approach to risk assessment, but again, 
having not formally studied their methdologies, I can not say if they’d do better with 
environmental risk assessment than does the EPA. 

Q6. Many EPA science activities are housed within regulatory offices. For example, 
EPA’s Office of Air and Radiation (rather than the Office of Research and Devel- 
opment), manages the National Fuel and Vehicle Emissions Laboratory, as well 
as the National Air and Radiation Environmental Laboratory. In your view, 
should science activities be organizationally insulated from regulatory activities 
to ensure objectivity and balance? 

A6. As I believe I mentioned in my testimony, it seems obvious to me that having 
too many related functions within one agency is a problem at EPA. Separation of 
functions such as risk assessment; peer-review of risk assessments; crafting of risk 
management plans; sponsoring of scientific inquiry related to risks the agency itself 
has a motivation to regulate; and cost-benefit analysis could only improve the qual- 
ity of our efforts to manage environmental risk. Just as we know that monopolies 
lead to reduced competitiveness and lower-quality products over time, the same is 
true for government agencies and entities. 

Q7. Some scientific information that is disseminated by federal agencies is subject 
to specific data quality requirements. Are there additional steps that you think 
could be taken to ensure that these peer review and data quality guidelines are 
followed or expanded for important scientific information at EPA? 

A7. In 2009, I testified before the Senate Environment and Public Works Com- 
mittee on the question of scientific integrity and transparency. This is what I told 
the Committee: 

As more and more of our nation’s public policy decisions involve the use of com- 
plex scientific information, it becomes more and more important that our policy- 
making institutions make use of such information in a process that is free of bias, 
is open to outside review and analysis, allows for the airing of divergent opinion, 
and is deliberative enough to ensure that the decisions we make are the right ones. 

As recent experience has shown, this is not currently the case. Policies intended 
to mitigate climate change and conventional pollution with the use of corn-ethanol 
have backfired badly. Rather than reduce greenhouse gas emissions, poorly-thought 
out ethanol mandates have increased them. Rather than reduce conventional air 
pollution, corn-ethanol has increased them, along with polluting surface and ground 
water, contaminating fish stocks with pesticide and herbicide residues, and expand- 
ing oceanic dead-zones caused by algae which bloom as they are over-fed by fer- 
tilizer run-off from corn agriculture. Most of these problems were raised by non-gov- 
ernmental analysts before the ethanol mandates were passed, but the policymaking 
process proved opaque to such cautionary voices. 

Now, warnings are coming from non-governmental policy analysts and scientists 
that we may see equally perverse impacts from other forms of renewable energy 
that are being promoted at breakneck speed through the spending of stimulus 
money, and pending legislation involving energy and climate change. For example, 
new scientific reports are validating concerns expressed by energy analysts that con- 
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centrated solar power systems may have unsustainable water demand and will im- 
peril fragile desert ecosystems. 

Warnings that wind turbines are not environmentally benign are being validated 
as they are found to cause noise pollution, visual blight, bird and bat kills, and po- 
tentially harm livestock. One recent study has found that mass transit systems may 
well produce more pollution than the automobiles and air travel they seek to dis- 
place. Left and right, we are seeing failings of our government’s policymaking bodies 
to listen to cautionary voices in the development of public policy dependent on the 
sound use of scientific information. 

The President’s memoranda on Transparency and Open Government and on Sci- 
entific Integrity are a good start, but they can only be considered a start in the proc- 
ess to ensure that scientific information is used properly in the process of public pol- 
icy formation. 

On the positive side of the ledger, the memoranda correctly identify certain impor- 
tant elements of a transparent process featuring scientific integrity. The President 
is exactly correct when he says that “political officials should not suppress or alter 
scientific or technological findings and conclusions.” 

It is also reassuring to see the President order that “To the extent permitted by 
law, there should be transparency in the preparation, identification, and use of sci- 
entific and technological information in policymaking.” 

Of particular importance, I think, is the President’s declaration that “Government 
should be participatory.” As the President observes, “Public engagement enhances 
the Government’s effectiveness and improves the quality of its decisions. Knowledge 
is widely disbursed in society, and public officials benefit from having access to that 
dispersed knowledge.” 

The President’s call for Executive departments and agencies to offer Americans 
greater opportunities to participate in policymaking processes and to infuse the deci- 
sion-making process with their “collective expertise and information” is spot on. 

But all too often, I have seen an assumption that only scientists working within 
government, or dependent on governmental grants have worthwhile knowledge to 
inject into public policy decision-making. There is, I believe, an inherent bias 
against scientists in the private sector, even though those are often the people who, 
day by day, in their laboratories, are producing the prescription drugs that save mil- 
lions, and who develop the technologies that empower billions. 

The same is true with regard to the President’s (and agency) emphasis on the 
peer-reviewed literature. As we have discovered through revelations about fraud in 
the scientific and medical literature, peer-review is no guarantee of accuracy. And 
often, the keys to publication are in the hands with those who have a vested interest 
in preserving the theory that gained them the prestige and standing to be consid- 
ered as peer-reviewers. As a recent article, ironically published in the peer-reviewed 
journal PLOS Medicine demonstrated, “most claimed research findings are wrong.” 

The President, Congress, and regulatory agencies should explicitly recognize that 
there is a legitimate role for non-governmental, independent scientific participation 
in the public policy decision-making process in terms of both personnel, and the in- 
jection of scientific research conducted outside the peer-reviewed literature. 

Many times, over my career, I have seen a lack of real opportunity for consulta- 
tion in the policymaking process. I have seen massive scientific reports issued by 
state and federal governmental agencies the day before Thanksgiving weekend, or 
just before the Christmas season, with minimal time allowed for the review of thou- 
sand-page scientific summary documents, and only trivial opportunities for mean- 
ingful consultation. We may see that again in coming months, where we’ve been 
promised the passage of landmark legislation on climate change, just in time for the 
Independence Day holiday, and many people’s summer vacation. 

Post-regulatory release of Regulatory Impact Assessments, as was the case with 
the 1997 revisions to the National Ambient Air Quality Standards, have sometimes 
made a mockery of the very idea of consultative decision making. 

Massive dockets in which thousands of review comments receive little more than 
blithe dismissals have been common features of governmental decision-making on 
important scientific issues I have sought to analyze over the last 18 years. 

Well-credentialed and experienced scientists have too often been frozen out of con- 
sultative processes because they are viewed as tainted by an industrial connection, 
or because they hold unorthodox views. 

In conclusion, the President’s memoranda on Transparency and Open Govern- 
ment, and Scientific Integrity are a good step, but only a single step in improving 
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the way that our government makes use of scientific information at all levels of the 
decision-making process. 

As more and more issues require the use of such information, more attention 
needs to be paid to reforming the processes by which scientific information is gath- 
ered, validated, balanced, summarized, and used to inform the decision-making 
process. 

Finally, it must always be remembered that science may be able to tell us “what 
is,” but it can never tell us “what to do.” Science informs — it does not compel. Public 
policy formation involves the balance of many factors, social, economic, ethics, eq- 
uity, individual rights, personal responsibility, and more. 

Creating openness and transparency in the scientific elements of the decision- 
making process is important, but that same level of openness, transparency, and 
consultation should infuse every element of the public policy development process. 

Q8. Nearly all of EPA’s recent Clean Air Act regulations have been justified on the 
basis of two studies that rely on entirely on data from the American Cancer So- 
ciety and the so-called Harvard Six Cities Study. Despite the fact that these data 
sets were developed with government funds and provide the basic Agency jus- 
tification for costly regulations, they are not publicly -available so they can be 
analyzed by other scientists. Do you support making this type of information 
transparent^ In your view, would making these underlying data sets available 
to everyone improve the Agency’s regulatory decisions? 

A8. While I understand the need to protect privileged information, and the con- 
fidentiality of the doctor-patient relationship, I believe that it is of utmost impor- 
tance that the key data used to determine risk-assessments be available for inde- 
pendent review: not simply by an institution picked by EPA, or by any given agency, 
but by anyone who wants to examine the data. 

As we have seen with climate change data manipulation in the case of the infa- 
mous hockey stick graph, and the revelation of a cliquish mentality on the part of 
researchers that was clearly to the detriment of our understanding of climate 
science, outside review is absolutely vital if we are to have confidence in the quality 
of the data that is being used to formulate far-reaching public policy initiatives. 

There is no reason why suitably blinded data could not be made available to allow 
others to review the validity of the ACS and Harvard 6-Cities studies which, as you 
point out, are overwhelmingly important in how EPA develops air quality policy. 
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Responses by Dr. Gary Marchant, 

Professor of Law and Executive Director 
Center for Law, Science & Innovation, 

Arizona State University 

Questions submitted by Subcommittee Chairman Andy Harris 

Ql. The Bipartisan Policy Center’s 2009 report, “Improving the Use of Science in 
Regulatory Policy” was mentioned several times during the hearing. This report 
made several suggestions that may be useful in guiding this Subcommittee’s ef- 
forts to reform regulatory science, including: 

• “Studies used in the formulation of regulation should be subject to data access 
requirements equivalent to those under the Data Access Act.” 

• “The process of conducting literature reviews” and “the process of naming advi- 
sory committees” should be made more transparent. 

• “Agencies should avoid turning repeatedly to the same scientists for service on 
advisory committees.” 

• Executive branch agencies need to “help clarify for both officials and the general 
public which aspects of disputes are truly about scientific results and which con- 
cern policy.” 

• “Policy makers should be wary of conclusions of risk that are expressed as a sin- 
gle number.” 

• “Federal agencies need to experiment with ways to increase the number of sci- 
entists who participate in peer review.” 

• “In presenting the conclusions of literature reviews, agencies and their scientific 
advisory committees need to be as open and precise as possible in discussing lev- 
els of risk and uncertainty.” 

Do you agree with any or all of these recommendations^ Do you have any additional 
comments or advice in pursuing these goals? 

Al. I agree with all these recommendations of the Bipartisan Policy Center’s report. 
I think the 4th recommendation listed on the importance of separating science from 
policy is critical. If the two are mixed together and confused, accountability and 
oversight, as well as public participation, are undermined. Putting scientific deter- 
minations in a separate institution from policy decisions, such as the Institute for 
Scientific Assessments I have proposed, would achieve this crucial separation. 

Q2. A recent joint report from the EPA’s Science Advisory Board and Board of Sci- 
entific Councilors recommended that the Agency “include sustainability in its re- 
search vision” in order to allow “EPA to adopt sustainability as a core principle 
to inform decisions and actions.” Is this emphasis on sustainability appropriate 
for EPA’s research and scienceactivities? 

A2. I have not read this report so cannot comment on it specifically. I do believe 
sustainability, if defined and applied broadly, can provide an appropriate framework 
for EPA’s research program. It would broaden EPA’s focus from solely environ- 
mental impacts to consider also broader economic and social values and impacts. It 
could help move EPA from the adversarial, command and control philosophy that 
was developed in the 1970s to a more collaborative, cooperative paradigm that is 
more appropriate to the more complex challenges facing us today. 

Q3. Many of the regulatory activities that EPA is currently undertaking are based 
upon statutes and priorities from several decades ago. In your view, are we fo- 
cusing our attention and scientific resources on the most pressing environmental 
issues? Are there ways that EPA could better prioritize? 

A3. The 1970s-era environmental statutes and regulations are increasing obsolete, 
and tend to shackle EPA into rigid, media specific, end of pipe controls that no 
longer represent the current priorities facing the nation. EPA should have more 
flexibility to explore market, cooperative/partnership, and other innovative ap- 
proaches to address remaining problems. For example, the EPA Project XL program 
was a win-win for the environment and companies subject to artificially rigid and 
narrow regulations. That program was terminated because it was not consistent 
with the outdated 1970s-era statutes EPA continues to regulate under. 

Q4. You proposed the establishment of an “Institute for Scientific Assessment” to con- 
duct R&D that is currently housed within EPA. Please describe which particular 
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science and R&D activities this Institute could take over from the Agency. In 
particular, please assess the feasibility of this organization overseeing and con- 
ducting: Integrated Risk Information System assessments; integrated science as- 
sessments for National Ambient Air Quality Standards; and determinations 
about research priorities. 

A4. The Institute for Scientific Assessments (ISA) as I have proposed it would not 
oversee research programs, but would conduct scientific assessments. The ISA 
would conduct assessments such as IRIS evaluations and NAAQS integrated sci- 
entific assessments. The results of these assessments would then be forwarded to 
EPA for its regulatory decisions. As I have proposed it, the ISA would not provide 
recommendations on research priorities, but it is conceivable that the mission of the 
ISA could be expanded in that direction. 

Q5. Many EPA science activities are housed within regulatory offices. For example, 
EPA’s Office of Air and Radiation (rather than the Office of Research and Devel- 
opment), marmges the National Fuel and Vehicle Emissions Laboratory, as well 
as the National Air and Radiation Environmental Laboratory. In your view, 
should science activities be organizationally insulated from regulatory activities 
to ensure objectivity and balanced 

A5. Yes, I believe scientific activities should be institutionally separated from regu- 
latory activities. Placing scientific assessments within the institutional context of, 
and under the control of, regulatory officials has the potential to consciously or sub- 
consciously influence and bias the nature and outcome of the scientific assessments. 
This is inconsistent with good scientific practice, which should be insulated from po- 
litical, policy and personal influences as much as practically possible. 

Q6. Some scientific information that is disseminated by federal agencies is subject 
to specific data quality requirements. Are there additional steps that you think 
could be taken to ensure that these peer review and data quality guidelines are 
followed or expanded for important scientific information at EPA? 

A6. I have two suggestions. The first one would be to place the activities subject 
to the peer review and data quality guidelines within an institutional context that 
is familiar with and dedicated to the principles of good scientific practice that are 
behind the guidelines. I think an organization like the Institute of Scientific Assess- 
ments that I have proposed, that is operated and managed from top to bottom by 
scientists applying scientific methods and customs, would be more likely to take se- 
riously and adhere to the peer review and data quality guidelines. My other sugges- 
tion is to make the guidelines enforceable through judicial review. Unless the guide- 
lines have teeth, they are unlikely to be influential in an environment with so many 
other factors influencing decision-making. 

Q7. INearly all of EPA’s recent Clean Air Act regulations have been justified on the 
basis of two studies that rely on entirely on data from the American Cancer So- 
ciety and the so-called Harvard Six Cities Study. Despite the fact that these data 
sets were developed with government funds and provide the basic Agency jus- 
tification for costly regulations; they are not publicly-available so they can be 
analyzed by other scientists. Do you support making this type of information 
transparent? In your view, would making these underlying data sets available 
to everyone improve the Agency’s regulatory decisions? 

A7. I definitely think the data should be transparent and publicly available. Trans- 
parency is a key requirement of science in order to allow scientific findings to be 
replicated. As one recent review of what makes good science stated: “The essence 
of good science is repeatability. Different scientists, in different places, at different 
times, can repeat good science if they follow the same methods and protocols.” Dr. 
Samuel McNaughten, What is Good Science?, Natural Resources & Envt. (ABA), 
Spring 1999, at 513. A recent special section in the journal Science on the impor- 
tance of data replication and reproducibility states: “Replication — the confirmation 
of results and conclusions obtained independently in another — is considered the sci- 
entific gold standard . . . The importance of replication and reproducibility for sci- 
entists is unquestioned. Sometimes attempts to replicate reveal scientific uncertain- 
ties. This is one of the main ways that sciences progresses. Unfortunately, in rare 
instances (compared to the body of scientific work), it can also indicate fraud.” B.R. 
Jasney et al.. Again, and Again, and Again . . . , Science 234:1225 (Dec. 2, 2011) (ci- 
tations deleted). If the underl 5 dng data are not made available, they cannot be rep- 
licated, and thus the scientific validity of the original study cannot be verified. 
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Purpose 

On Friday, February 3, 2012, the Subcommittee on Energy and Environment of 
the Committee on Science, Space, and Technology will hold a second day of testi- 
mony to provide external perspectives on the need to reauthorize and reform 
science, research, and development activities at the Environmental Protection Agen- 
cy (EPA); explore the intersection of Agency-supported science and its regulatory 
mission; and receive focused recommendations to raise the level, quality, usefulness, 
and objectivity of EPA science, including any necessary changes to the Environ- 
mental Research, Development and Demonstration Authorization Act. 

Witnesses 

• Mr. Daniel Greenbaum, President and Chief Executive Officer, Health Effects 
Institute 

• Dr. Deborah Swackhamer, Professor, Environmental Health Sciences, Uni- 
versity of Minnesota, and Chairwoman, EPA Science Advisory Board 

• Mr. Michael Walls, Vice President, Regulatory and Technical Affairs, Amer- 
ican Chemistry Council 

• Dr. Richard Belzer, President, Regulatory Checkbook 

• Dr. Jerald Schnoor, Allen S. Henry Chair in Engineering, Department of 
Civil and Environmental Engineering, University of Iowa 

• Dr. S. Stanley Young, Assistant Director for Bioinformatics, National Institute 
of Statistical Sciences 

Background 

The Environmental Research, Development, and Demonstration Authorization Act 
(ERDDA) authorizes research and scientific activities at the Environmental Protec- 
tion Agency (EPA). Originally enacted in 1976, Congress subsequently passed an- 
nual authorizations through fiscal year 1981. In addition to establishing annual au- 
thorization levels, these statutes also directed EPA policy in a variety of 
areas,including establishing the Office of Research and Development (ORD), ^ re- 
quiring a five-year environmental R&D plan, and creating EPA’s Science Advisory 
Board (SAB). 


Year 

Act 

Public Law Number 

1976 

ERDDA 

94-475 

1977 

ERDDA on978 

95-155 

1978 

ERDDA of 1979 

95-477 

1979 

ERDDA or 1980 

96-229 

1980 

ERDDA of 1981 

96-569 


See Appendix 1 for EPA organizational structure. 
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Since 1981, there have been a number of bills introduced to reauthorize ERDDA 
that were not ultimately enacted into law. ^ As a result, explicit authorization of 
EPA’s environmental R&D ended at the end of fiscal year 1981. This failure to com- 
prehensively reauthorize EPA research, development, and demonstration programs 
and activities illustrates a broader trend among expired environmental statutes. 
The Congressional Research Service notes this trend, stating, “Although Congress 
somewhat recently has renewed the authorization of appropriations for certain EPA 
programs and activities through targeted amendments to various statutes, a more 
comprehensive reauthorization of many of the statutes that EPA administers has 
not been enacted for a number of years.” ^ 

In addition to ERDDA, EPA also derives authority for R&D activities through 
other major environmental statutes. For example, under the Clean Air Act, the EPA 
Administrator must issue criteria that “accurately reflect the latest scientific knowl- 
edge useful in indicating the kind of extent of all identifiable effects on public health 
or welfare which may be expected from the presence of such pollutant in the ambi- 
ent air.”'' Through the Safe Drinking Water Act (SDWA), EPA sets standards based 
on “the best available, peer-reviewed science and supporting studies conducted in 
accordance with sound and objective scientific practices.”® Similarly, the Clean 
Water Act (CWA) requires EPA to publish water quality information “accurately re- 
flecting the latest scientific knowledge.”® 

In many cases, these major regulatory statutes also authorize specific R&D pro- 
grams and activities. For example, the Clear Air Act established a national research 
and development program for the prevention and control of air pollution including 
establishing technical advisory committees and research on air pollutant moni- 
toring. The SDWA authorized the Administrator of EPA to conduct research and 
studies relating to the causes, diagnosis, treatment, control, and prevention of phys- 
ical or mental diseases resulting directly or indirectly from contaminants in the 
water including improved methods to identify and measure contaminants in drink- 
ing water and improved methods to identify and measure the health effects of con- 
taminants in drinking water. The CWA directed the Administrator to establish na- 
tional programs for the prevention, reduction, and elimination of pollution and as 
part of such programs to work in cooperation with other State and federal agencies 
to coordinate and accelerate research, investigation, experiments, demonstrations, 
and studies relating to the causes, effects, extent, prevention, reduction, and elimi- 
nation of pollution in the navigable waters of the U.S. 

The science enterprise at EPA is spread across program offices and regions. ORD 
is organized into three national labs (comprised of 18 separate labs) and four na- 
tional centers (which have 19 divisions). In addition to 18 labs within ORD, there 
are nine labs split among several program offices and each of the 10 regions has 
its own lab.® In FY 2010, the appropriations level for EPA Science and Technology 
activities (S&T includes ORD and the other 19 labs) was $874.9 million. The appro- 
priations level for FY 2011 was $840.3 million. The FY 2012 House Committee- 
passed appropriations level is $777.6 million, and the FY 2012 Senate Committee 
draft appropriations level is $809 million. 

The fragmented nature of EPA R&D presents a challenge to program manage- 
ment and coordination and has complicated efforts to evaluate the effectiveness of 
these activities. Numerous studies conducted by the EPA Office of Inspector Gen- 
eral, the Government Accountability Office, and others have cited significant con- 
cerns with the science activities of the Agency and the difficulties in evaluating the 
usefulness of the science to program needs. These studies have offered recommenda- 
tions on how to improve the science enterprise at EPA, but many of these rec- 
ommendations have not been implemented. 


2 HR 3115 (1982), HR 2804 (1982), S. 1205 (1982), S. 2577 (1983), HR 2899 (1984), S. 1292 
(1984), HR 2319 (1985), S. 2702 (1985), S. 1144 (1986), HR 2355 (1987), HR 1523 (1987), HR 
2153 (1989), HR 4873 (1990), HR 2404 (1991), S. 1655 (1991), HR 1994 (1993), S. 1545 (1993), 
HR 2405 (1995), HR 1814 (1995), HR 3322 (1996), HR 1276 (1997), HR 1742 (1999), HR 1743 
(1999). 

2 Congressional Research Service, “Environmental Laws: Summaries of Major Statutes Admin- 
istered by the Environmental Protection Agency,” RL30798, August 11, 2011. 

'‘42 U.S.C. §7408(a)(2) (2000). 

5 42 U.S.C. §300g-l(b)(3)(A)(i). 

5 33 U.S.C. 11314(a)(1). 

2 See Appendix 2. 

5 See Appendix 3. 
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The bottom of this chart shows three national labs and four national centers. The three national labs are broken down in 
Appendix 3. 
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Chairman Harris. The Subcommittee on Energy and Environ- 
ment will come to order. 

Good morning. Welcome to day two of the hearing entitled “Eos- 
tering Quality Science at EPA: Perspectives on Common Sense Re- 
form.” 

In front of you are packets containing the written testimony, bi- 
ographies and Truth in Testimony disclosures for today’s witness 
panel. 

I want to welcome everyone, and we will have five minutes for 
opening statements by myself and the ranking member. I want to 
welcome everyone to the second day of our hearing on fostering 
quality science at EPA. As this is a continuation of the hearing 
held on November 30th of last year, I will be brief. 

Unfortunately, the Environmental Research, Development, and 
Demonstration Authorization Act, or ERDDAA, which is the stat- 
ute authorizing R&D at EPA as well as the Science Advisory 
Board, was last reauthorized for fiscal year 1981. I think we can 
all agree that our fiscal, environmental, and economic priorities 
have changed dramatically over the past 30 years, and we should 
have statutes and a Congressional role in environmental policy 
that reflects those changes. As we have held nearly a dozen over- 
sight hearings on specific EPA issues during this Congress, we 
have seen patterns of behavior that suggest the need for significant 
reforms. 

At day one of this hearing, we received testimony from several 
witnesses with decades of experience with the Agency: Susan Dud- 
ley of George Washington University, who formerly served as head 
of the White House Office of Information and Regulatory Affairs; 
Alan Moghissi of the Institute for Regulatory Science; Ken Green 
of the American Enterprise Institute; and Gary Marchant of Ari- 
zona State University. They provided specific recommendations on 
reforming scientific activities at EPA, including the need to sepa- 
rate science and policy; to quantify uncertainties; to ensure greater 
transparency in the data, models, and assumptions used in regu- 
latory decisions; to prioritize environmental problems and solu- 
tions; and to stop overly alarmist approaches to benefit-cost anal- 
ysis. 

I want to thank the witnesses for appearing before the Sub- 
committee, and I look forward to continuing this important con- 
versation with this panel of experts. That concludes my opening 
statement. 

[The prepared statement of Mr. Harris follows:] 

Prepared Statement of Chairman Andy Harris 

I want to welcome everyone to the second day of our hearing on “Fostering Qual- 
ity Science at EPA: Perspectives on Common Sense Reform.” 

As this is a continuation of the hearing held on November 30th of last year, I will 
he brief. 

Unfortunately, the Environmental Research, Development, and Demonstration 
Authorization Act, or ERDDAA (ERDA), which is the statute authorizing R&D at 
EPA as well as the Science Advisory Board, was last reauthorized for fiscal year 
1981. I think we can all agree that our fiscal, environmental, and economic prior- 
ities have changed dramatically over the last 30 years, and we should have statutes 
and a Congressional role in environmental policy that reflects these changes. As we 
have held nearly a dozen oversight hearings on specific EPA issues during this Con- 
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gress, we have seen patterns of behavior that suggest the need for significant re- 
forms. 

At day one of this hearing, we received testimony from several witnesses with dec- 
ades of experience with the Agency: Susan Dudley of George Washington University, 
who formerly served as head of the White House Office of Information and Regu- 
latory Affairs; Alan Moghissi of the Institute for Regulatory Science; Ken Green of 
the American Enterprise Institute; and Gary Marchant of Arizona State University. 
They provided specific recommendations on reforming scientific activities at EPA, 
including the need to separate science and policy; to quantify uncertainties; to en- 
sure greater transparency in the data, models, and assumptions used in regulatory 
decisions; to prioritize environmental problems and solutions; and to stop overly 
alarmist approaches to benefit-cost analysis. 

I want to thank the witnesses for appearing before the Subcommittee, and I look 
forward to continuing this important conversation with this panel of experts. 

Chairman Harris. Mr. Miller. 

Mr. Miller. Thank you, Chairman Harris. 

Today the Subcommittee, as Chairman Harris said, meets again 
for part two of the hearing we held at the end of November on 
science at the EPA. The first two hearings in this series were a dis- 
appointment, and a missed opportunity to build a helpful record in 
preparation for the reauthorization of the Environmental Research, 
Development, and Demonstration Authorization Act. Mercifully, 
there is an acronym: ERDDAA. 

However, today, I am pleased to see that we have some panelists 
with the experience and knowledge required to address in detail 
critical improvements that can make EPA’s research enterprise 
more effective, efficient, and transparent. At the least, this is not 
just a panel of witnesses armed only with talking points and flail- 
ing criticism meant to undermine or dismantle the one agency 
charged with protecting our citizens and the environment from un- 
lawful pollution. Let us use their time and ours wisely. 

As I have said before, I approach this task hoping to work with 
my Republican counterparts in pursuing reforms that will lead to 
better research practices that help EPA accomplish its mission. 
While we will not always agree on the best way to do that, I am 
not interested in restructuring EPA to take the only environmental 
cop off the beat. There are legitimate concerns related to EPA’s re- 
search infrastructure and processes, but they are complex, and we 
have to approach the process in a well-thought out and planned 
manner. 

I have authored and co-authored many bills in my time here. I 
understand the amount of research, stakeholder conversations, and 
thought that must take place to write legislation as important and 
ambitious as the reauthorization of ERDDAA. 

EPA’s scientific research is increasingly important as we seek to 
understand and address more complex environmental issues that 
continue to emerge and evolve. That was demonstrated just 48 
hours ago when this Subcommittee met to consider EPA’s role in 
examining groundwater research and the start of the Pavillion 
Study process. 

Scientific research knowledge and technical information are fun- 
damental to EPA’s mission, and to inform its standard-setting, reg- 
ulatory, compliance, and enforcement functions. That is why Con- 
gress created advisory bodies such as the Clean Air Scientific Advi- 
sory Committee and the Science Advisory Board that were created 
to provide independent advice on the science that allows the Ad- 
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ministrator to make regulatory decisions. In addition to advice 
from an array of experts from many fields, the scientific process 
also involves the use of epidemiology and modeling to aid in hazard 
identification, which is only the first stage of quantitative risk as- 
sessment. 

But in the scientific process, epidemiology and modeling inves- 
tigations are not the only approach to research studies. It is a mul- 
tidisciplinary approach that includes real-time monitoring, clinical 
and laboratory studies, model development, measurement and ex- 
posure methods, characterization of sources, and control tech- 
nologies. Just like the process we need in reauthorizing ERDDAA, 
the responsibility of the scientific process and regulatory decision 
making takes a host of perspectives, methods, and techniques. 

In short, science should inform and support the decisions we 
make, that Congress makes and the EPA makes, and most impor- 
tant, we all have an ultimate responsibility to do everything we can 
to make sure that everyone enjoys a decent quality of life. 

And Chairman Harris, I yield back. 

[The prepared statement of Mr. Miller follows:] 

Prepared Statement of Ranking Member Brad Miller 

Thank you, Chairman Harris. Today the Subcommittee meets again for part two 
of the hearing we held at the end of November on science at the EPA. The first two 
hearings in this series were a disappointment and a missed opportunity to build a 
helpful record in preparation for the reauthorization of the Environmental Research, 
Development, and Demonstration Authorization Act, or ERDDA. 

However, today I am pleased to see that we have some panelists with the experi- 
ence and knowledge required to address in detail critical improvements that can 
make EPA’s research enterprise more effective, efficient, and transparent. At the 
least, this is not just a panel of witnesses armed only with talking points and flail- 
ing criticism meant to undermine or dismantle the one agency charged with pro- 
tecting our citizens and the environment from unlawful pollution. Let’s use their 
time and ours wisely. 

As I have stated before, I approach this task hoping to work with my Republican 
counterparts in pursuing reforms that will lead to better research practices that 
help EPA accomplish its mission. While we will not always agree on the best way 
to do that, I am not interested in restructuring EPA to take the only environmental 
cop off the beat. 

There are legitimate concerns related to EPA’s research infrastructure and proc- 
esses, but they are complex, and we have to approach this process in a well-thought- 
out and planned manner. I have authored and co-authored many bills in my time 
here. I understand the amount of research, stakeholder conversations, and thought 
that must take place to write legislation as important and ambitious as the reau- 
thorization of ERDDA. 

EPA’s scientific research is increasingly important as we seek to understand and 
address more complex environmental issues that continue to emerge and evolve. 
This was demonstrated just 48 hours ago when this Subcommittee met to consider 
EPA’s role in examining ground-water research and the start of the Pavilion Study 
process. 

Scientific research, knowledge, and technical information are fundamental to 
EPA’s mission, and inform its standard-setting, regulatory, compliance, and enforce- 
ment functions. That is why Congress created advisory bodies such as the Clean Air 
Scientific Advisory Committee (CASAC) and the Science Advisory Board (SAB) that 
were created to provide independent advice on the science which allows the Admin- 
istrator to make regulatory decisions. In addition to advice from an array of experts 
from many fields, the scientific process also involves the use of epidemiology and 
modeling to aid in hazard identification, which is only the first stage of quantitative 
risk assessment. 

But in the scientific process that epidemiology and modeling investigations are 
not the only approach to research studies. It is a multidisciplinary approach includ- 
ing real-time monitoring, clinical and laboratory studies, model development, meas- 



79 


urement and exposure methods, characterization of sources, and control tech- 
nologies. Just like the process we need in reauthorizing ERDDA, the responsibility 
of the scientific process and regulatory decision making takes a host of perspectives, 
methods, and techniques. 

In short, science should inform and support the decisions we make. And most im- 
portant, we all have an ultimate responsibility to do everything we can to ensure 
that EVERYONE continues to enjoy a decent quality of life. 

With that. Chairman Harris, I yield back. 

Chairman Harris. Thank you very much. 

At this time I would like to introduce our witness panel and 
thank each of them for appearing before us today. 

Our first witness is Mr. Daniel Greenbaum, President and CEO 
of the Health Effects Institute. He has been a member of the U.S. 
National Research Council, Board of Environmental Studies and 
Toxicology, and Vice Chair of its Committee for Air Quality Man- 
agement. Mr. Greenbaum has over three decades of governmental 
and non-governmental experience in environmental health. 

Our next witness is Dr. Deborah Swackhamer, a Professor of En- 
vironmental Health Sciences at the University of Minnesota and 
the Chairwoman of the EPA Science Advisory Board. She is also 
a Governor appointee on the Minnesota Clean Water Council and 
recently completed a one-year term as President of the National In- 
stitute of Water Resources. She is also a member of the Editorial 
Advisory Board for the Journal Environmental Science and Tech- 
nology and is Chair of the Editorial Advisory Board of the Journal 
of Environmental Monitoring through the end of last year. 

Our third witness is Mr. Michael Walls, Vice President of Regu- 
latory and Technical Affairs for the American Chemistry Council. 
He has experience in a wide range of domestic chemical regulatory 
issues including the Toxic Substance Control Act, Emergency Plan- 
ning and Community Right to Know Act, and the Resource Con- 
servation and Recovery Act. His experience also includes work on 
international chemical regulatory issues, including the Europe 
Commission’s Registration, Evaluation and Authorization of 
Chemicals. 

Our fourth witness is Dr. Richard Belzer, President of Regulatory 
Checkbook. From 1988 to 1998, Dr. Belzer served as Staff Econo- 
mist in OMB’s Office of Information and Regulatory Affairs, where 
he reviewed major federal regulations and the supporting analyses. 
He focused primarily on the assessment of benefits and opportunity 
costs, both of which involve crucial links to human health risk as- 
sessments. 

Our fifth witness is Dr. Jerald Schnoor, Allen S. Henry Chair in 
Engineering in the Department of Civil and Environmental Engi- 
neering at the University of Iowa. Dr. Schnoor chaired the Board 
of Scientific Counselors for the Environmental Protection Agency, 
Office of Research and Development, from 2000 to 2004. While 
serving as Editor in Chief of Environmental Science and Tech- 
nology, he guided the leading journal in the world in both environ- 
mental engineering and environmental science. 

Our sixth and final witness is Dr. S. Stanley Young, the Assist- 
ant Director for Bioinformatics at the National Institute of Statis- 
tical Sciences. He is a fellow of the American Statistical Association 
and the American Association for the Advancement of Science. He 
has authored or co-authored over 50 papers including six Best 
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Paper Awards and a highly cited book, Resampling Based Multiple 
Testing. 

Thank you all for appearing before the Subcommittee today. As 
our witnesses should know, spoken testimony is limited to five min- 
utes each, after which the Members of the Committee will have five 
minutes each to ask questions. 

I now recognize our first witness, Mr. Daniel Greenbaum of the 
Health Effects Institute. 


STATEMENT OF MR. DANIEL GREENBAUM, 
PRESIDENT AND CHIEF EXECUTIVE OFFICER, 

HEALTH EFFECTS INSTITUTE 

Mr. Greenbaum. Mr. Chairman and Members of the Committee, 
I am pleased to appear before you today. I am Dan Greenbaum, as 
you said, from the Health Effects Institute. We are a nonprofit re- 
search institute with joint and balanced funding from tJ.S. EPA 
and industry that for over 30 years has produced trusted science 
to inform air quality decisions. I draw on HEFs and other experi- 
ences to highlight five important principles for producing credible 
science. 

HEI was born out of controversy between EPA and industry over 
whose science could be believed. We were established as an inde- 
pendent, nonpartisan entity to produce health science that could be 
agreed to by all parties, and we are designed with several key ele- 
ments to ensure impartiality: an independent, high-level board of 
directors of distinguished science and policy leaders who are not 
from sponsors; standing committees of independent, respected ex- 
perts; a research committee to oversee all research; and a separate 
review committee to conduct intensive peer review of all research 
and prepare a commentary on the findings. The board also ap- 
points special expert committees to conduct targeted re-analyses of 
key studies and systematic reviews of the literature. 

All results from HEI are published and available for free, and we 
work to actively provide access to all underlying data. And impor- 
tantly, HEI produces policy-relevant science, but we do not take 
policy positions. 

Now, HEI was not established to replace all science produced for 
air quality and other environmental decisions but HEFs design was 
designed to produce science of the highest quality and credibility 
in often controversial circumstances. Five key principles guide this 
work. 

First, we engage scientists who are independent and objective, 
scientists from a wide variety of arenas, not just environment and 
health. It is essential that public and private science organizations 
actively reach out to the widest possible range of scientists with di- 
verse perspectives and skills. Now, scientist recruitment must also 
avoid real or readily perceived conflicts of interest, and HEI, the 
National Academies, EPA and others have procedures in place to 
identify these, but these reviews of conflicts should not become a 
straitjacket that, for example, disqualifies well-qualified scientists 
just because they have been funded at times by industry or EPA 
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or an environmental organization. We need the best science, as 
long as it is objective, from wherever it can come. 

Second, science should be funded through vigorous, open competi- 
tion. HEI and a number of others including U.S. EPA’s Star Grant 
program, which has received exemplary reviews from the National 
Research Council, use well-established techniques for soliciting, re- 
viewing, scoring and selecting projects. Again, the broad-based re- 
cruitment of scientists to compete and participate in these proc- 
esses is essential to ensuring a level playing field for the widest 
possible sets of institutes and science perspectives. 

Third, we need to apply the full range of multidisciplinary skills, 
drawing on experts in emissions, exposure, toxicology and epidemi- 
ology, but perhaps most important, we at HEI have placed the field 
of statistics at the center of our work, insisting on predesigned sta- 
tistical analysis plans and subjecting each study’s results to de- 
tailed statistical review to make sure that the best techniques were 
applied. 

Fourth, all results must be subjected to intense peer review and 
re-analysis if needed. Now, peer review has been a cornerstone of 
science for many years, but with the profusion of scientific journals 
in recent years, the quality of peer review can vary substantially. 
This is further complicated by the tendency of some journals to be 
more interested in publishing studies that have found a positive ef- 
fect, or so-called publication bias. HEFs peer review process re- 
quires a comprehensive report of all findings, not only positive re- 
sults, a broad-based standing expert peer review panel which has 
had nothing to do with the study, the active engagement of at least 
two statisticians in each review, and the ability to request and gain 
access to all underlying data as part of that review. 

And finally, science should be conducted and reported with full 
transparency. HEI seeks to produce its work with the widest de- 
gree of disclosure of results and underlying data. This is critical to 
ensuring that all positive and negative results are reported and 
that the broader science community can access and further analyze 
the results and data. We have even placed entire databases for our 
studies on the Web for anybody to go to at any time when that was 
possible. 

Thank you for this opportunity to testify. I would be pleased to 
answer any questions. 

[The prepared statement of Mr. Greenbaum follows:] 

Prepared Statement of Mr. Daniel Greenbaum, 

President and Chief Executive Officer, 

Health Effects Institute 


Summary 

The production of high-quality, credible science is of critical importance to inform- 
ing often-controversial policy decisions on environment and health. For over 30 
years, the Health Effects Institute, an independent, not-for-profit research institute 
with joint and balanced funding from U.S. EPA and industry, has produced trusted 
science in a variety of forms to inform air quality decisions. This testimony draws 
from that experience — and from the results of the recent report of the Bipartisan 
Policy Center (Improving the Use of Science in Regulatory Policy) — to highlight key 
principles of producing credible science, including: 

• Engaging scientists who are independent and objective; 
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• Funding science through vigorous open competition', 

• Applying the full range of multi-disciplinary skills', 

• Subjecting all results to intense peer review, and re-analysis if needed; and 

• Conducting and reporting science with full transparency. 

This testimony describes how each of these key principles contributes to producing 
credible science; the critical elements necessary for applying them successfully, and 
the degree to which practice at US EPA and elsewhere in government includes these 
approaches currently and/or could be enhanced. 

Testimony 

Mr. Chairman, and Members of the Committee, it is my pleasure to appear before 
you at this important hearing, “Fostering Quality Science at EPA: Perspectives on 
Common Sense Reform — Day II.” I am Daniel S. Greenbaum, President of the 
Health Effects Institute (HEI), an independent, not-for-profit research institute with 
joint and balanced funding from U.S. EPA and industry that, for over 30 years, has 
produced trusted science in a variety of forms to inform air quality decisions. I also 
was pleased to serve recently on the Committee of the Bipartisan Policy Center on 
Science and Policy, a multi-party expert panel that made recommendations ^ on im- 
proving the development and use of science in policy. I draw on the rich experience 
of HEI, and the recommendations of the Science and Policy Committee to highlight 
several important principles for producing credible science to inform environment 
and health decisions. 

The Health Effects Institute 

HEI was born out of controversy. During implementation of the Clean Air Act 
rules for air quality and vehicle emissions in the 1970s, there was substantial dis- 
agreement between manufacturers and the U.S. EPA about the underlying health 
science driving decisions. HEI was established with the support of U.S. EPA and 
industry as an independent, non-partisan entity to produce health science that could 
be agreed to by all parties — and could serve as the basis for better decisions. HEI 
is designed with several key elements to ensure its impartiality: 

• Joint and balaneed eore funding from US EPA and industry, 

• An independent, high-level Board of Directors of distinguished science and policy 
leaders to guarantee the integrity of the science, with members agreed to by the 
EPA Administrator and industry but not containing any current sponsor em- 
ployees; 

• Standing Committees of subject matter experts in exposure, toxicology, epidemi- 
ology, statistics and other disciplines who are not employees of sponsors and 
who may not have demonstrated “a lack of objectivity” in their field: 

• Research Committee to design, conduct competitions for, and oversee all re- 
search; 

• Review Committee to conduct intensive peer review of all HEI-funded re- 
search, and prepare a Commentary on the scientific findings and their impli- 
cations for decisions. 

• Special Expert Committees appointed according to the same principles to con- 
duct targeted reanalyses of key studies and systematic reviews of the literature 
in important areas. 

• Full transparency, with all results published and available for free electroni- 
cally, and active provision of access to underlying data; 

• Importantly, HEI produces policy-relevant science, hut does not take policy posi- 
tions. 

With these elements in place, HEI has funded over 250 studies of a wide range 
of air pollutants; reanalyses of a number of epidemiologic studies central to deci- 
sions; and special reviews of the literature on diesel exhaust, air toxics, traffic ef- 
fects, and more. HEI’s work has been widely accepted as credible and comprehensive 
and is regularly cited in decision making in the U.S. and worldwide. 


1 Bipartisan Policy Center. 2009. Improving the Use of Science in Regulatory Policy. Wash- 
ington, DC: Bipartisan Policy Center. 
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Principles of Credible Science 

HEI was not established to replace all science produced for air quality policy deci- 
sions. Much science was then, and is today, produced directly with funding from 
U.S. EPA, the National Institutes of Health, and others. But HEI’s design was de- 
veloped to produce science of the highest quality and credibility at the most critical 
and often controversial junctures of science and decisions, and the key principles 
that HEI has applied can inform the enhancing of credibility of all science produced 
for informing decisions. These key principles are: 

• Engaging scientists who are independent and objective: Quality science for deci- 
sions requires the active involvement of a wide range of talented individuals 
from diverse perspectives. Many scientists are fully engaged in their research 
and teaching and hesitant to become overly involved in often-controversial 
science/policy settings. One result of this is that at times one can find a range 
of scientists actively engaged in the work of organizations like the National 
Academy of Sciences, but despite the best recruitment efforts of entities such 
as the Science Advisory Board, unwilling to engage in the scientific work of 
agencies like EPA. To further enhance skills, HEI has sought to engage sci- 
entists from a wide variety of arenas, not just environment and health; it is es- 
sential that public and private science organizations actively reach out to the 
widest possible range of scientists, seeking consciously to engage scientists with 
diverse perspectives and skills. 

For maximum credibility, scientist recruitment must also ensure that scientists 
do not carry with them real or readily perceived conflicts of interest, e.g., a di- 
rect hnancial interest in the outcome of the scientihc deliberation. The BPC 
Science and Policy Report systematically reviews the many detailed approaches 
that have been adopted by U.S. EPA, other federal agencies, the NAS, and oth- 
ers for identifying both biases and conflicts of interest, and recommends en- 
hanced approaches to this important task. 

It is important, however, that such reviews of bias and conflicts not act to unneces- 
sarily place scientist selection in a “strait jacket” that, for example, disqualihes well- 
qualified scientists simply because they have been funded by industry or U.S. EPA, 
or have done work or work currently for industry or an environmental organization. 
Some of the best experts have received funds from a range of sponsors, are capable 
of providing a balanced perspective on the science, and should be included unless 
there is a real and current conflict of interest. 

• Funding science through vigorous open competition: A hallmark of the highest- 
quality science is to ensure that it is selected and funded through the highest 
levels of peer-reviewed competition. HEI and a number of other research pro- 
grams, including U.S. EPA’s STAR grants program (which has received exem- 
plary reviews from the National Research Council) have used well-established 
techniques for soliciting, reviewing, scoring, and selecting such projects. At the 
same time, this is an area where the broad-based recruitment of scientists to 
participate in these selection processes, and the recusal of scientists from re- 
viewing applications from their own institutions, is essential to ensuring a 
“level playing held” for competitors from the widest possible set of institutions 
and scientihc perspectives. 

• Applying the full range of multi-disciplinary skills: Since its inception, HEI has 
seen fully multi-disciplinary science as the only way to answer complex ques- 
tions facing decision makers in environmental health. Thus, for example, a 
team studying the health effects of certain emissions, or peer reviewing the re- 
sults of such a study, must include engineering and exposure measurement ex- 
pertise. And the best health studies will draw on a combination of toxicological 
and epidemiological techniques to determine whether a certain exposure is hav- 
ing an effect. Perhaps most important, HEI has placed the held of biostatistics 
at the center of its work, insisting on pre-designed statistical analysis plans for 
each major project, and subjecting each study’s results to intense statistical re- 
view to ensure that (a) the best and most appropriate statistical techniques 
were applied and (b) any positive results (i.e., those showing an “effect”) are 
placed in the context of the full range on positive and negative results before 
interpreting the study’s conclusions. 

• Subjecting all results to intense peer review, and re-analysis if needed: Peer re- 
view has been a cornerstone of science for generations and has served well, in 
general, to identify the strongest contributions to the scientihc literature on a 
wide variety of topics. However, with the profusion of scientihc journals in re- 
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cent years, and the diversification of peer review processes, the degree to which 
any particular journal article is subjected to the highest level of peer review can 
vary substantially. This is further complicated by the tendency of journals to 
be more interested in publishing studies that have found a positive “effect,” a 
“publication bias” which has now been documented in a number of settings. ^ 

The HEI peer review process was designed to address these shortcomings, espe- 
cially for science at critical intersections between science and decisions. That 
process includes several key elements: (a) a comprehensive report of all find- 
ings, not necessarily only the “positive” results; (b) a broad-based standing 
panel of experts (the HEI Review Committee) which has had nothing to do with 
the study and meets in person to review each report and to prepare a detailed 
Commentary on the study findings and their implications; (c) the active engage- 
ment of at least two biostatisticians in each review; and (d) the contractual abil- 
ity to request and gain access to all underlying data generated in the study and 
used in the analysis. These and other steps result in a level of peer review that 
is widely regarded as being as intense as, and in some cases more intense than, 
the peer review at the best scientific journals. 

HEI has, at times, also been asked by Congress, U.S. EPA, industry, and others 
to go beyond its intensive peer review of its own studies to play two other in- 
tense review roles: the reanalysis of key studies that are particularly central to 
decisions (e.g., the HEI reanalysis of the Harvard Six Cities and American Can- 
cer Society studies), ^ and the systematic review of the complete scientific lit- 
erature on emissions, exposure, and health (e.g., recent reviews of the science 
on the potential effects of exposure to air toxics and to traffic-generated air pol- 
lution). In each of these cases HEI’s Board of Directors appoints multi-discipli- 
nary expert panels according to the same principles of independence to oversee 
reanalysis and systematic literature reviews. And those efforts are then in turn 
subjected to high levels of peer review by experts who have not previously been 
involved. 

• Conducting and reporting science with full transparency: From its inception, 
HEI has sought to produce its work with the widest degree of disclosure of re- 
sults and underlying data. This is critical to ensuring that all results — both 
positive and negative — are reported, and that the broader science community 
can fully access, and further analyze, the results and data. HETs comprehensive 
reports present, for free Web distribution, all methods and results, along with 
the Commentary of the HEI Review Committee. And since the mid-1990s, HETs 
Board of Directors has had in place a Data Access Policy that has both encour- 
aged HEI investigators to make their data and analysis freely available on the 
Web (for example, the data underlying HETs National Morbidity, Mortality and 
Air Pollution Study (NMMAPS)), ® and to facilitate — wherever HEI investigators 
have full ownership of underlying data — access for other investigators to the 
data. 

Conclusions — Toward Credible Science for Environment and Health Decisions 

In conclusion, it is clear that science can and should play an important role in 
providing the foundation for decisions on environment and health, and that to do 
so, the science needs to be of the highest quality and credibility. U.S. EPA and other 
agencies have established procedures to produce and review science for decisions, 
and in many cases those procedures work to enhance the quality and credibility of 
the science. The HEI experience, founded out of a desire by both industry and U.S. 
EPA for more readily trusted science, has illustrated a number of key principles 
that can lead to even better science for decisions in the years to come. Thank you 


^Cf. Samet J.M., Zeger S.L., Dominici F., Curriero F., Coursac I., Dockery D.W., Schwartz J., 
Zanobetti A. 2000. The National Morbidity, Mortality, and Air Pollution Study, Part II: Mor- 
bidity and Mortality from Air Pollution in the United States. Research Report 94. Health Effects 
Institute, Cambridge, MA. 

^Krewski D., Burnett R.T., Goldberg M.S., Hoover K., Siemiatycki J., Jerrett M., 
Abrahamowicz M., White W.H. 2000. Reanalysis of the Harvard Six Cities Study and the Amer- 
ican Cancer Society Study of Particulate Air Pollution and Mortality. A Special Report of the 
Institute’s Particle Epidemiology Reanalysis Project. Health Effects Institute, Cambridge, MA. 

’^Cf. HEI Panel on the Health Effects of Traffic-Related Air Pollution. 2010. Traffic-Related 
Air Pollution: A Critical Review of the Literature on Emissions, Exposure, and Health Effects. 
HEI Special Report 17. Health Effects Institute, Boston, MA. 

® www.ihapss.jhsph.edu. 
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for this opportunity to testify. I would be pleased to answer any questions the Com- 
mittee may have. 

Chairman Harris. Thank you very much. 

I now recognize our second witness, Dr. Deborah Swackhamer of 
the University of Minnesota and Chairwoman of the EPA Science 
Advisory Board. 

STATEMENT OF DR. DEBORAH SWACKHAMER, PROFESSOR, 
ENVIRONMENTAL HEALTH SCIENCES, 

UNIVERSITY OF MINNESOTA, 

AND CHAIRWOMAN, EPA SCIENCE ADVISORY BOARD 

Dr. SWACKHAMMER. Good morning, Chairman Harris, Ranking 
Member Miller and distinguished Committee Members. My name 
is Deborah Swackhamer, and I hold the Denny Chair in Science 
Technology and Public Policy at the Humphrey School of Public Af- 
fairs at the University of Minnesota. I am trained as an environ- 
mental chemist, and I am Professor of Environmental Health 
Sciences in the School of Public Health and co-direct the Univer- 
sity’s Water Resources Center. 

I was appointed Chair of the SAB in 2008 by EPA Administrator 
Stephen Johnson and reappointed for a second term in 2010 by Ad- 
ministrator Lisa Jackson. While my perspectives and opinions are 
my own, I am testifying today on behalf of the SAB. 

The SAB provides science advice to the EPA Administrator on a 
wide range of scientific and technical issues. These issues are com- 
plex, and they require a diversity of experience to address. The 
SAB membership brings expert knowledge from the natural and 
physical sciences, engineering, health sciences, and social sciences 
including economics. Based on my years of service on the board, I 
believe that the agency has a robust process for identifying mem- 
bers with outstanding scientific credentials who are committed to 
helping improve the quality of agency science. The SAB and its 
committees and panels review agency work products, undertake 
special studies when requested, and perform self-initiated studies 
on topics that the Board considers to be of critical importance. 

Recent SAB advice that is directly relevant to this hearing in- 
cludes two reports we produced in 2009 and 2010 on strategic di- 
rections for EPA research prepared for ORD to encourage ap- 
proaches and strategies needed to do their science most effectively. 
These two reports have been instrumental in moving ORD’s re- 
search enterprise towards a more interdisciplinary approach and 
one that can respond more nimbly and effectively to the needs of 
the program offices and the regional offices. The Administrator’s 
One EPA and ORD Assistant Administrator Anastas’s “The Path 
Forward” strategies are consistent with our previous advice. The 
board is in the process of finalizing a report on how the agency can 
do a better job of integrating science and problem formulation in 
its decision making. 

The SAB is supportive of many changes that have taken place 
in ORD in recent years. More could be done, more is being done, 
but I believe and our reports have indicated that ORD is moving 
in the right direction. We have advised strongly for an integrated 
approach to EPA’s scientific research, and the agency has re- 
sponded, as indicated by the realignment of research programs 
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from 13 independent programs to six integrated programs. We have 
advised to include a greater degree of social and decision science 
research, and the agency is moving to fill this need. We have ad- 
vised to develop the capacity to respond to emerging issues and the 
new program structure should move them in the right direction. 
We have advised the agency to partner more nationally and inter- 
nationally and develop truly collaborative research efforts in these 
times of limited and shrinking resources, and they have been cre- 
ative in doing so. Finally, we urged ORE) to support and creative 
incentives for their scientists to be more innovative, and they have 
responded. 

The SAB and presumably this Subcommittee share the goal and 
commitment to assist EPA in producing and using high-quality 
science to protect human health and the environment. The best 
available science is essential to sound decision making but is not 
the only aspect of sound policy decisions. What is best available 
science? While hard to provide a simple one-size-fits-all definition, 
generally, it is scientific results, conclusions, and technical informa- 
tion that has been produced using proven methods that has been 
peer reviewed where hypotheses are tested with objective and unbi- 
ased approaches and that has the support for its conclusions from 
other independent studies. The role of the SAB is to examine the 
scientific and technical knowledge that was synthesized within the 
agency on a given issue and provide advice as to whether this 
science was appropriate and adequate for its intended use. 

Finally, the letter from Chairman Harris requested that I com- 
ment on the capability of EPA to conduct and use the best avail- 
able science to fulfill its mission. The agency certainly has the ca- 
pability, given its excellent scientific enterprise. It is sorely short 
of resources to provide the capacity needed for all the science ques- 
tions at the agency, and yet, there is no other agency where such 
environmentally focused and directed science is being done to fill 
the unique mission of protecting the public’s health and the envi- 
ronment on which they depend. 

Investing in EPA science is a wise investment. That said, this ca- 
pability would be improved by continuing to address scientific ques- 
tions from an interdisciplinary approach, by partnering more cre- 
atively with others, by involving stakeholders in problem formula- 
tion, and integrating science across the agency for the most effec- 
tive decision making. 

Thank you for the opportunity to speak with you today. 

[The prepared statement of Dr. Swackhamer follows:] 

Prepared Statement of Dr. Deborah Swackeiamer, Professor, 
Environmental Health Sciences, University of Minnesota, 
and Chairwoman, EPA Science Advisory Board 

My name is Deborah Swackhamer, and I hold the Charles M. Denny, Jr., Chair 
in Science, Technology, and Public Policy in the Hubert H. Humphrey School of Pub- 
lic Affairs at the University of Minnesota, and co-direct the University’s Water Re- 
sources Center. I am trained as an environmental chemist and am also professor 
of Environmental Health Sciences in the School of Public Health. 

I was appointed chair of the SAB in 2008 by EPA Administrator Stephen Johnson 
and reappointed for a second term in 2010 by EPA Administrator Lisa Jackson. 
From 2006-2008, I served on the Board of Scientific Counselors (BOSC) for EPA’s 
Office of Research and Development (ORD). While my views, perspectives, and opin- 
ions are my own, I am testifying at this hearing on behalf of the SAB. 
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The Role of the SAB 

The SAB provides science advice to the EPA Administrator on a wide range of 
scientific and technical issues. These issues are complex and require a diversity of 
expertise to address. The SAB membership brings expert knowledge from the nat- 
ural and physical sciences, engineering, health sciences, and social sciences includ- 
ing economics. Based on my years of service on the Board, I believe that the Agency 
has a robust process for identifying members with outstanding scientific credentials 
who are committed to helping improve the quality of Agency science. The SAB and 
its committees and panels review Agency work products, undertake special studies 
when requested, and perform self-initiated studies on topics that the Board con- 
siders to be of critical importance. The Board is in the process of finalizing a report 
on how the Agency can do a better job of integrating science and problem formula- 
tion in its decision making. i 

Recent SAB advice that is directly relevant to this hearing includes the two re- 
ports we produced (2009, 2010)2 on Strategic Directions for EPA Research, prepared 
for ORD to encourage approaches and strategies needed to do their science most ef- 
fectively. These two reports have been instrumental in moving ORD’s research en- 
terprise towards a more interdisciplinary approach, and one that can respond more 
nimbly and effectively to the needs of the Program Offices and Regional Offices. The 
Administrator’s “One EPA” and ORD Assistant Administrator Anastas’ “The Path 
Eorward” strategies are consistent with our previous advice. lA^ 

The SAB is supportive of many changes that have taken place in ORD in recent 
years. We have advised strongly for an integrated approach to EPA’s scientific re- 
search, and the Agency has responded, as indicated by its realignment of research 
programs from 13 individual programs to six integrated programs. We have advised 
to include a greater degree of social and decision science research, and the Agency 
is moving to fill this need. The social sciences are a needed component to adequately 
address issues such as sustainability, homeland security, risk communication, valu- 
ation, and environmental stewardship and human behavior. The Agency needs to 
develop a strategy for developing this capability. We have advised to develop capac- 
ity to respond to emerging issues, and the new program structure should move them 
in that direction. We have advised the Agency to partner more nationally and inter- 
nationally and develop truly collaborative research efforts in these times of limited 
and shrinking resources, and they have been creative in doing so. Einally, we urged 
ORD to support and create incentives for their scientists to be more innovative, and 
they have created a highly successful internal program for Innovation Grants and 
have modified their internal rewards system to encourage the best scientific publica- 
tions. 

To summarize, we are supportive of these changes at ORD. More could be done, 
more is being done, but I believe, and our reports have indicated, that ORD is mov- 
ing in the right direction. 

Quality, Usefulness and Objectivity of EPA Science — the Role of SAB 

The SAB, and presumably this Subcommittee, share the goal and commitment to 
assist EPA in producing and using high-quality science to protect human health and 
the environment. The best available science is essential to sound decision making 
but is not the only aspect to sound policy decisions. What is “best available science”? 
While hard to provide a simple one-size-fits-all definition, generally it is scientific 
results, conclusions, and technical information that has been produced using proven 
methods, that has been peer reviewed, where hypotheses are tested with objective 
and unbiased approaches, and that has support for its conclusions from other inde- 
pendent studies. EPA cannot possibly do all of the science needed by the Program 
Offices and Regional Offices. Some of this needed science is conducted within EPA, 
and some science is used from outside research to verify, supplement, and in general 
add to the collective body of knowledge used to inform a given decision. 

The role of the SAB is to examine the scientific and technical knowledge that was 
synthesized within the Agency for a given issue, and provide advice as to whether 
this science was appropriate and adequate for its intended use. In SAB reviews of 
EPA science assessments, we consider whether the data, reports, and other re- 
sources used were peer reviewed and compared and contrasted appropriately. It is 


^Science Integration for Decision Making at the U.S. Environmental Protection Agency (EPA) 
(draft January 5, 2012). 

^ EPA’s Strategic Research Directions 2008: An Advisory by the EPA Science Advisory Board. 
EPA-SAB-09-006. 

^Office of Research and Development Strategic Research Directions and Integrated 
Transdisciplinary Research. EPA-SAB- 10-010. 
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my understanding that EPA has clear guidance regarding peer review of its own sci- 
entific work, and for data quality and transparency."* For purposes of maximum 
transparency and quality assurance, we usually advise the Agency not to include 
reports that have not been peer reviewed, or journal manuscripts in preparation or 
draft form but not yet published. 

As a researcher who has received funding from EPA and many other agencies, 
I have found that EPA has very high standards for data quality and assurance. 

Enhancing EPA Science 

Finally, the letter from Chairman Harris requested that I comment on the capa- 
bility of EPA to conduct and use the best available science to fulfill its mission. The 
Agency certainly has the capability given its excellent scientific enterprise. It is 
sorely short of resources to provide the capacity needed for all the science questions 
at the Agency, and yet there is no other agency where such environmentally focused 
and directed science is being done to fill the unique mission of protecting the 
public’s health and the environment on which they depend. Investing in EPA 
science is a wise investment. That said, this capability would be improved by con- 
tinuing to address scientific questions from an interdisciplinary approach, by 
partnering more creatively with others, by involving stakeholders in problem formu- 
lation, and integrating science across the Agency for the most effective decision 
making. 

Thank you for the opportunity to speak to you today. 

Chairman Harris. Thank you very much. 

I now recognize our third witness, Mr. Michael Walls of the 
American Chemistry Council. 

STATEMENT OF MR. MICHAEL WALLS, 

VICE PRESIDENT, REGULATORY AND TECHNICAL AFFAIRS, 
AMERICAN CHEMISTRY COUNCIL 

Mr. Walls. Good morning, and thank you very much for the op- 
portunity to provide this testimony on behalf of the American 
Chemistry Council. 

Now, the business of chemistry is fundamentally the business of 
science. The chemical industry practices high-quality science to fos- 
ter the discovery of new chemistries and the development of new 
tools by which we can assess the hazards, uses, and exposures of 
chemicals. We similarly expect high-quality science and reliable as- 
sessment procedures to underpin effective and efficient regulatory 
decisions by the government. 

Now, my testimony today boils down to a very simple message: 
the process for bringing science to bear in regulatory and policy de- 
cision making at EPA and at other federal agencies is broken. The 
quality of the science has suffered as a result, and the credibility 
and reliability of the decisions made on the basis of that science is 
at stake. Now, Congress, the agencies, the industry and the Amer- 
ican public have a significant interest in using the best science to 
ground those decisions. The fact is that science and the government 
are reasonably likely to lead the regulatory decisions, and those de- 
cisions have practical implications for businesses. State, and local 
governments and individuals. 

I would just like to focus on several examples drawn from EPA’s 
IRIS program as well as some other government programs. The 
IRIS draft assessment on n-Butanol relies on two studies deter- 
mined to be unreliable by the Organization for Economic Coopera- 


'^Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of In- 
formation Disseminated by the Environmental Protection Agency. EPA/260R-02-008. 
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tion and Development in a review that was sponsored by yet an- 
other office within EPA. There is no indication that that conflict is 
going to be resolved. The National Academy of Sciences directed 
EPA to do nonlinear modeling in support of its IRIS assessment of 
dioxin. Eive years later, EPA published yet another draft of the as- 
sessment that similarly failed to do the nonlinear modeling that 
was requested. EPA Science Advisory Board justly criticized the 
draft for that failure. The National Toxicology Program, part of the 
Department of Health and Human Services, issued its 12th report 
on carcinogens in July 2011. The report makes many of the same 
errors in its assessment of formaldehyde that the National Acad- 
emy criticized EPA for in its own review of formaldehyde. That 
12th report also viewed styrene and came to a sharply different 
conclusion than a 2010 evaluation by another division within HHS. 

Why do we need to get this right? Well, in the IRIS case, it is 
particularly important because 80 percent of IRIS assessments 
haven’t been updated in more than 15 years. Ninety percent are at 
least ten years old. Meanwhile, the science that informs our under- 
standing of chemicals and of exposures has continued to advance 
by leaps and bounds. That new science should surely inform our 
regulatory and policy decisions. 

The Eederal Government’s processes for assessing risk lack a 
consistent, coherent framework. That framework should bind the 
agencies to an appropriate and transparent approach to weigh the 
evidence, consider uncertainty, and keep up with advances in the 
field. Peer review is a critical step to ensure a high level of quality 
and reliability. Despite recommendations from the NAS and from 
the SAB, little has been done to ensure that peer review is con- 
sistent within and among the federal agencies. In short, we need 
to modernize and streamline these processes to meet both today’s 
needs and our future challenges. 

My written testimony outlines some recommendations for im- 
proving the quality and process of science in three areas. Number 
one, establishing sound risk assessment procedures, standards and 
criteria; two, enhancing peer review; and three, leveraging the 
emerging science and technology to reach better decisions. 

The chemical industry looks forward to working with this Sub- 
committee in its continuing effort to improve science and risk as- 
sessment in the government. I very much appreciate the invitation 
to join the discussion today and look forward to your questions. 

[The prepared statement of Mr. Walls follows:] 

Prepared Statement of Mr. Michael Walls, 

Vice President, Regulatory and Technical Affairs, 

American Chemistry Council 


Summary 

The American Chemistry Council (ACC) ^ very much appreciates this opportunity 
to provide testimony on common-sense measures to foster quality science at the En- 
vironmental Protection Agency (EPA) and throughout the Federal Government. 


^The American Chemistry Council (ACC) represents the leading companies engaged in the 
business of chemistry. ACC members apply the science of chemistry to make innovative products 
and services that make people’s lives better, healthier, and safer. ACC is committed to improved 

Continued 



90 


The business of chemistry is fundamentally the business of science. This business 
of science is a critical component for manufacturing safe products required to house, 
feed, and protect people in the United States as well as provide for the tremendous 
quality of life experienced by American citizens who enjoy many high-quality and 
safe consumer goods that were unavailable just a few decades earlier. ACC member 
companies rely on science to conduct the research necessary to discover new chem- 
istries and identify new applications of existing chemistries. They also rely on 
science to develop new tools for assessing the potential hazards, exposures, and 
risks of chemical substances. As one of the Nation’s most regulated industries, ACC 
member companies similarly expect high-quality science — and reliable assessment 
processes — to underpin effective and efficient regulatory decisions by the Federal 
Government. 

Unfortunately, processes for conducting and reviewing chemical assessments at 
EPA and other government agencies are not always based on the consistent use of 
the best available science. The lack of scientific quality and reliability directly com- 
promises societal access to cost-effective and safe products that house, feed, and pro- 
tect us while making life more enjoyable at the same time. While there has been 
much recent focus on EPA’s Integrated Risk Information System (IRIS), the prob- 
lems identified by the National Academy of Sciences (NAS) in the IRIS program are 
also evident in other government chemical assessment programs. 

EPA has acknowledged many of the deficiencies in the IRIS program and is tak- 
ing some welcome steps to address the concerns identified by the NAS. EPA is also 
making an important effort to develop and evaluate emerging technologies to im- 
prove chemical assessments, and ACC has been pleased to support these efforts. 

ACC’s testimony today outlines a number of recommendations to improve the 
quality and process of science at EPA and more broadly through the Federal Gov- 
ernment. The following areas should receive particular attention: 

• Improving the quality of science through sound risk assessment processes, 
standards and criteria. 

• Improving the quality of science through enhanced peer review. 

• Enhancing the quality of science by leveraging emerging science and technology. 

I. Improving the Quality of Science Through Sound 
Risk Assessment Processes, Standards, and Criteria 

The Subcommittee’s inquiry into the level, quality, usefulness, and objectivity of 
science at the Environmental Protection Agency (EPA) is timely. There are well- 
known deficiencies in EPA’s Integrated Risk Information System (IRIS) — defi- 
ciencies that Congress has directed the National Academy of Sciences to review. But 
the problems that affect the Agency’s ability to assure that the science generated, 
reviewed, and used is of the highest quality are not unique to EPA. ^ ACC’s testi- 
mony today outlines a number of recommendations to improve the quality and proc- 
ess of science at EPA and more broadly through the Federal Government. 

At the heart of the problem in the Federal Government’s processes for assessing 
risks to environment and human health is the lack of a consistent, coherent, 
science-based framework that binds the agencies to an appropriate and transparent 
approach for weighing evidence, considering uncertainty, and keeping up with ad- 
vances in the field. The processes for considering scientific information and data and 


environmental, health, and safety performance through Responsible Carer, common-sense advo- 
cacy designed to address major public policy issues, and health and environmental research and 
product testing. The business of chemistry is a $674 billion enterprise and a key element of the 
Nation’s economy. It is one of the Nation’s largest exporters, accounting for 10 cents out of every 
dollar in U.S. exports. Chemistry companies are among the largest investors in research and 
development. It is also one of the Nation’s most heavily regulated industries. 

2 EPA’s Integrated Risk Information System (IRIS) has been the focus of much critical atten- 
tion recently. As the Subcommittee is well aware, the National Academy of Sciences (NAS) has 
expressed concern over “[t]he persistence of limitations of the IRIS assessment methods and re- 
ports, particularly in light of the continued evolution of risk-assessment methods and the grow- 
ing societal and legislative pressure to evaluate many more chemicals in an expedient manner.” 
The NAS report further cites a lack of clarity and transparency as a “repeating theme” over 
the last decade, insufficient documentation on methods and criteria for identifying evidence from 
relevant studies, and a lack of information useful in assessing the weight of the evidence, among 
other problems. These concerns are not limited to IRIS, or even EPA. For example, the Report 
on Carcinogens (RoC) issued by National Toxicology Program (NTP), housed in the Department 
of Health and Human Services (HHS). The 12th RoC, released in July 2011, makes many of 
the same methodological errors in its evaluation of formaldehyde as IRIS did in its review, and 
the 12th RoC’s review of styrene conflicts with a 2010 evaluation by another HHS entity. Simi- 
lar concerns exist with EPA’s Clean Air Scientific Advisory Committee. 
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the standards and criteria used in risk assessment need to be modernized and 
streamlined to meet both today’s needs and greater challenges of the future. 

A. Integrated Risk Information System (IRIS) 

Despite continued evolution of the EPA IRIS process, specific fundamental im- 
provements to the program are necessary to ensure that IRIS assessments devel- 
oped by EPA are firmly based on up-to-date scientific knowledge, meet the highest 
standards of scientific inquiry and integrity, and are evaluated in accordance with 
acceptable scientific approaches. 

IRIS is used by EPA as the primary source of information regarding the potential 
adverse human health effects of chemicals. IRIS is also a leading source of health 
risk information for other federal. State, and international regulatory bodies. Given 
the importance that IRIS evaluations have for EPA program offices, other federal 
agencies, and State governments, as well as their impacts on the private and public 
sectors, it is clear that significant improvements are warranted and long overdue. 

Many of these necessary improvements were outlined in Chapter 7 of the April 
2011 NAS scientific peer review report on formaldehyde and underscored during two 
recent Congressional oversight hearings on IRIS. Despite general agreement with 
the need to make the changes recommended by the NAS, EPA has yet to provide 
further details on how it will implement the NAS IRIS improvements. The U.S. 
Government Accountability Office (GAO) has called on EPA to develop a clear plan 
for fixing IRIS. ® 

In an effort to move EPA in this direction. Congress recently passed bipartisan 
legislation which directs EPA in FY 2012 to: 

• Incorporate, as appropriate, the recommendations of Chapter 7 of the National 
Research Council’s Review of the Environmental Protection Agency’s Draft IRIS 
Assessment of Formaldehyde into the IRIS Process; and 

• Issue a progress report to House and Senate Committees on Appropriations and 
relevant Congressional authorizing committees no later than March 1, 2012, de- 
scribing its implementation of the National Research Council’s Chapter 7 rec- 
ommendations for ongoing and new assessments. 

This action by Congress rightfully underscores the widespread agreement that 
more work is needed to improve IRIS so the program delivers scientifically defen- 
sible assessments. 

EPA does not need to go back to square one to improve IRIS and the assessments 
already underway. But more than a cursory review and more than simple improve- 
ments are needed. In particular, EPA should determine whether all ongoing assess- 
ments — including those that the Agency is revising to take into account peer review 
and public comments — meet the NAS standards for reviewing studies, evaluating 
weight of evidence, determining mode of action, establishing cause and effect, and 
for selecting the dose-response method for quantifying potential health risks. If an 
IRIS assessment falls short, it must be upgraded. 

ACC firmly believes that this process can be accomplished without undue delay 
in making IRIS assessments final. All stakeholders have an interest in IRIS assess- 
ments that rely on the best available scientific information regarding hazard and 
exposure; employ consistent, objective methods and models; utilize transparent eval- 
uation procedures for data quality, cause and effect; and that weigh the full body 
of scientific evidence. If an ongoing IRIS assessment does not meet these criteria 
(for example, if a draft IRIS assessment does not employ a robust weight-of-the-evi- 
dence approach), the program must accept that more time will be needed to get the 
assessment right. The credibility of the IRIS program is not enhanced by assess- 
ments that fail to address the basic criteria for quality and reliability. 

Importantly, there is nothing in the current IRIS program that provides an incen- 
tive for companies to develop new data and information and to use new toxicological 
methods and tools to generate and gather that data. Indeed, the industry has little 
confidence that new informationand data can overcome the conservative default as- 
sumptions employed in the program or the persistent problems identified in peer re- 
view. 

In ACC’s view, two principal solutions can help meet the Federal Government’s 
need to enhance chemical risk assessment, and to restore credibility in the results. 
First, federal agency standards for risk assessment need to be updated. Ideally, the 
same set of updated standards would apply across the Federal Government. There 


3 Grovernment Accountability Office, “Chemical Assessments: Challenges Remain With EPA’s 
Integrated Risk Information System Program,” GAO-12-42, Dec 9, 2011. Available at http: I I 
www.gao.gov / assets 1 590 / 586620.pdf. 
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are a variety of ways this might be accomplished. Second, the laws and rules gov- 
erning scientific peer reviews should be updated to make that vital process more ef- 
fective and transparent. 

B. Improved Standards for Risk Assessment 

Under existing authority, there is a clear role for the Office of Management and 
Budget (0MB) in reviewing agency assessments and coordinating a robust inter- 
agency review to promote uniformity in process and results. It is clear that federal 
risk assessment activities are not being coordinated, despite direction and guidance 
provided by 0MB bulletins and memoranda. * Moreover, there is no current govern- 
mentwide oversight to ensure coordination. As a consequence, the lack of a coordi- 
nated approach to these various assessment programs creates the potential for du- 
plication and inconsistent findings. Most troubling, each federal agency conducting 
such assessments does so in a different way, using different processes and stand- 
ards. 

To address this lack of coordination and consistency, federal agencies need to 
adopt updated state-of-the-art standards for human health and environmental risk 
assessments. Ideally, agencies would all follow a consistent set of standards. Agen- 
cies should be required to explain how they followed these standards, including pro- 
viding a clear articulation of reasons for choices they made in the process. Agency 
compliance with those requirements would be enhanced if it were subject to regular 
oversight, including judicial review. 

Federal standards for risk assessment should: 

• Include criteria for evaluating the validity of test methods and the reliability 
and credibility of data. 

• Require an assessment of the weight of evidence regarding hazard and expo- 
sure, based on criteria that should include elements such as a systematic review 
of all relevant and reliable toxicological, epidemiological, and mechanistic data, 
including negative results; a preference for human data, where it is relevant 
and adequate; and consideration of biologically plausible modes of action most 
relevant to humans. 

• Require agencies to present the distribution of estimated hazards or risks, in- 
cluding central tendency values. 

• Require agencies to characterize uncertainty and variability quantitatively, 
where feasible, and to explain these and other limitations of the analysis with 
sufficient clarity to be understood by non-scientists. 

• Require full disclosure of: 

• Data, methods and models sufficient to allow independent reanalysis by 
qualified experts; 

• Rationales for choosing key studies, methods and models; 

• Assumptions, extrapolations and policy judgments; 

• Plausible alternatives and related impacts; and 

• Major risk conclusions and degree of confidence based on uncertainties. 

• Outline a process of stakeholder engagement, including: 

• An interactive “problem formulation” at the outset of each assessment to 
identify key issues and data needs; 

• Timing assessments to make maximum use of relevant external research; and 

• Outreach regarding proposed charge questions for peer review of the assess- 
ment. 

• Consider how the concept of proportionality can be addressed in risk assess- 
ment standards, so that risk assessments are more closely linked to the decision 
they are used to justify. 

There are a number of options by which these standards can be developed and 
appropriate oversight of Agency adherence to the standards established. ® For exam- 
ple, if the Environmental Research, Development, and Demonstration Authorization 


'^OMB “Final Information Quality Bulletin for Peer Review” and OMB’s “Updated Principles 
for Risk Analysis” (http: j lwww.whitehouse.gov I sites ! default t files ! omb ! memoranda j fy2005 j 
m05-03.pdf; http: / / www. whitehouse.gov j sites ] default I files ! omb I assets ! regu- 

latory-matters -pdfj m07-24.pdf). 

^ These proposals do not address who is responsible for generating the data that is used in 
these assessments. ACC assumes that companies will typically have that responsibility. 
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Act (ERDDA) is reauthorized, Congress can direct EPA to develop and implement 
these standards. 

Congress could also consider a mandate that federal agencies collaborate in an 
interagency committee that ■would be tasked with developing risk assessment stand- 
ards that all agencies would have to follow. This might include standards outlining 
the basic assumptions underlying risk assessment methodologies (such as concepts 
of threshold versus linear modeling), the use of animal data, and weight of the evi- 
dence approaches. The logic behind this approach is that it could bring together the 
agencies charged with balancing competing risks and benefits from protective inter- 
ventions (e.g., the Food and Drug Administration, the Centers for Disease Control) 
with those agencies whose mandates are to reduce risks (e.g., EPA and the National 
Toxicology Program). The critical point, of course, is to avoid the development of 
lowest common denominator standards that simply preserve the status quo. 

Congress could also direct the practice of federal agency risk assessment across 
the Federal Government by requiring the Office of Management and Budget (0MB) 
and the Office of Science and Technology Policy (OSTP) to develop standards broad- 
ly applicable across the government. Both 0MB and OSTP have career staff knowl- 
edgeable about risk assessment and are interested in improving agency estimates 
of risks. In 2006, 0MB and OSTP issued a proposed bulletin providing guidance to 
federal agencies regarding their conduct of health, safety, and environmental assess- 
ments. ® Ultimately, 0MB reinforced existing guidance, and noted an expectation 
that agencies would follow the principles. Unfortunately, agencies appear to rou- 
tinely ignore these principles. Upper- and lower-bound estimates are not provided, 
negative studies are not discussed, and the uncertainties and limitations of the as- 
sessment are not articulated. Congress should ensure that agencies follow these 
basic principles. 

II. Improving the Quality of Science Through Enhanced Peer Review 

Integrating scientific methods across EPA and the federal agencies also requires 
enhancing the manner in which the broader scientific community is engaged in the 
assessment process. In ACC’s view, the standards governing scientific peer reviews 
should be updated to make this vital process more effective. Peer engagement and 
review are two critical factors in the effort to ensure high-quality, reliable science 
supports decision making. Although ACC focuses on EPA in this section, the rec- 
ommendations we provide should inform enhanced peer review across the govern- 
ment. 

Independent peer review is a critical element of EPA’s scientific policies and prac- 
tices, and to date has received less attention than other elements of IRIS. Peer re- 
view is defined by EPA as “an in-depth assessment of the assumptions, calculations, 
extrapolations, alternate interpretations, methodology, acceptance criteria, and con- 
clusions pertaining to the specific major scientific and/or technical work product and 
of the documentation that supports them.”'^ Peer review plays a crucial role in de- 
velopment of the best scientific evaluation and is integral to identifying information 
that would reduce uncertainty in significant areas of the assessment. The process 
of peer review should be structured to accomplish these objectives. There are several 
areas to consider for enhancing EPA’s peer review process: 

• Peer review panels need to have sufficient time and resources to fulfill their re- 
sponsibilities. 

• Rather than base peer review charge questions solely on the input provided by 
the lead agency office, the preparation of these charge questions should reflect 
stakeholder input and be developed using an iterative process. Development of 
the charge questions should be initiated at the problem formulation step, and 
then issued as a refined draft coinciding with the release of the draft IRIS as- 
sessment. Public comments on the draft charge questions should be solicited. 

• Peer review charge questions should be written in order to facilitate objective 
consideration of alternative plausible scientific views rather than from the van- 
tage point of giving deference to the interpretation presented in the Agency as- 
sessment. This pro'vides peer reviewers greater opportunity to consider alter- 
native scientific views such as those offered by stakeholders. 

• As recommended in the Bipartisan Policy Center’s report “Improving the Use 
of Science in Regulatory Policy,” EPA should “explicitly differentiate between 


«See 71 Fed. Reg. 2600 (Jan. 17, 2006). 

■'U.S. Environmental Protection Agency, Peer Review Handbook 3rd Edition, EPA/lOO/B-06/ 
002, at 12. Available at http:j I www.epa.gov / peerreview I pdfs I peer -review -handbook -2006.pdf. 



94 


questions that involve scientific judgments and questions that involve judg- 
ments about economics, ethics and other matters of policy.”® 

• Peer review meetings should be structured to encourage open scientific dialogue 
and thoughtful scientific deliberation. Stakeholder input should not be limited 
to a few minutes at the beginning of a meeting; greater effort should be made 
to structure the meetings so that stakeholder input is provided and deliberated 
at strategic times throughout the meeting. Moreover, peer reviewers should not 
be dissuaded from embarking on open technical discussion/ scientific exchange 
with stakeholders. 

• In selecting peer review panel members, the foremost consideration should be 
given to expertise. Qualified scientists from industry should be given equal con- 
sideration for appointment based on the subject matter, and in accordance with 
applicable conflict-of-interest provisions. There is unanimity among the most 
authoritative sources on this point, including the National Academies of Science 
and the Society of Toxicology: 

Appointments to scientific advisory bodies should be based principally on the 
scientific credentials, demonstrated accomplishments, and professional credi- 
bility of the nominee. His/her source of employment and funding (past or 
present), religious beliefs, political persuasion, sexual orientation, gender, or 
race/ethnicity should not be used as (a) determinant(s) of exclusion to such a 
scientific advisory body. ® 

The Office of Government Ethics (OGE) has issued detailed rules under the Ethics 
in Government Act (EGA) and the federal criminal code addressing conflict of inter- 
est, and impartiality, on the part of government employees, including “Special Gov- 
ernment Employees” serving part-time on peer review committees. Fairly inter- 
preted, the EGA and those rules strike a fair balance and allow persons employed 
by industry or non-governmental organizations to serve as reviewers in many cases. 
However, agencies have tended to interpret these rules in ways that (i) restrict the 
participation of industry personnel and (ii) are too accepting of persons who are not 
really independent of the agency or the work being reviewed. Congress may wish 
to revisit the EGA and the rules, and their role in promoting high-quality, reliable 
science. 

In ACC’s view, EPA’s Science Advisory Board (SAB) has adopted generally sound 
processes and criteria for peer review of Agency action. There is room for improve- 
ment, however. For example, the SAB should ensure that the SAB peer reviewers 
fully understand their independent roles as peer reviewers. At times, however, it ap- 
pears that peer reviewers are overly deferential to EPA, reluctant to be seen as 
criticizing EPA staff. It also appears that EPA staff have an unfettered ability to 
comment throughout the peer review meetings, and their constant presence may 
have a chilling effect on frank and open discussion among the peer reviewers. This 
practice contrasts sharply with NAS peer reviews. 

ACC is generally encouraged by EPA’s recent announcement that it will establish 
a standing SAB panel for IRIS assessments. Assuming that that standing panel is 
truly independent, and the panel process addresses the concerns such as the role 
of EPA staff and how review comments are incorporated into completed IRIS assess- 
ments, this approach could help promote a more reliable and consistent IRIS proc- 
ess. 

Responding to peer review and public comments is another area where the Agency 
needs to make improvements in its practices. It is imperative that the Agency pro- 
vide a robust response in writing to comments as part of the assessment revision 
process that follows the publiccomment and peer review phases. Where the Agency 
elects not to address a peer review finding or recommendation, or a significant pub- 
lic comment, EPA should provide a written justification. This practice should be 
made routine for all federal agencies. 

The current practice of having the same office that develops the assessment draft 
the charge questions, review public and peer review comments, decide which rec- 
ommendations and findings to act on and which to ignore, and develop the final as- 
sessment is clearly not a best practice. The inherent value of peer review — indeed 
the inherent value of EPA’s SAB — is to provide an objective, robust scientific review 
of the agency’s scientific work product. ACC believes there is value in having an 
“honest broker” to oversee and ensure that the Agency adequately revises assess- 


®The Center’s report is available at http:! I www.bipartisanpolicy.orgl sites ! default (files j 
BPC%20Science%20Report%20fnl.pdf. 

^ See Society of Toxicology, Appointment and Participation of Scientists on Peer Review Panels 
and Scientific Advisory Boards, available at http://www.toxicology.org/pmlAdvisoryBoard.asp. 
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ments in a manner that addresses both public comments and the findings and rec- 
ommendations of independent scientific peer review. At this time, upon receiving a 
SAB or NAS panel report, EPA unilaterally decides what elements to accept or re- 
ject — a practice that clearly has not worked, particularly given the NAS report on 
formaldehyde. Reviewing bodies should have an opportunity to address how the 
Agency intends to implement the recommendations. 

III. Improving the Quality of Science by Leveraging Emerging Science and 
Technology 

One of ACC’s key objectives is to ensure that federal risk assessment policies and 
practices rely on 21st century knowledge of toxicology, biological modes of action, 
and advanced mechanistic technologies. There are dramatic changes underway in 
the science and technology of assessing chemical risks. These changes promise a 
revolution in the speed and accuracy with which chemical hazards, exposures, and 
risks are evaluated and managed. 

While EPA has made important investments in developing new, highly reliable 
technologies that can speed chemical assessments, not all offices within EPA appear 
disposed to adopt these technologies when appropriate. Successful integration of 
emerging science and technology into risk assessment will require a concerted and 
methodic approach to evaluate the science and build consensus around their readi- 
ness. 

The field of toxicology has grown more sophisticated as we have learned more 
about the biochemical mechanisms of toxicity and the differences between humans 
and test animals. New and exciting technologies for evaluating chemicals are emerg- 
ing. In some cases, however, agencies are not well prepared to implement these new 
tools. Many federal agencies still cling to a set of conservative default assumptions 
little changed from the 1960s and ’70s, and appear to be reluctant to adopt new 
technologies. 

In ACC’s view, it is critical that the Federal Government and the chemical indus- 
try be actively engaged in the transformation of chemical safety sciences. ACC mem- 
ber companies have made a significant, continuing investment in the ACC Long- 
range Research Initiative (LRI) to inform and advance this objective. ACC currently 
commits some $5 million annually to the program, which is designed to help: 

• Drive development of innovative approaches to assess and interpret health risks 
from low-dose exposures to chemicals and exposures to mixtures. 

• Develop and apply new tools to interpret the explosion of biomonitoring and 
high-throughput testing data regarding human health risks. 

• Accelerate the shift away from traditional high-dose animal toxicological testing 
by developing, validating, and promoting broad acceptance of approaches with 
greater relevance for humans. 

• Translate emerging research outcomes for decisions about the safety of our 
chemicals by partnering with thought leaders from industry, government, aca- 
demia, and public interest groups. 

The LRI program’s hallmark is the collaborative work to catalyze technological in- 
novations in chemical safety sciences with the Federal Government, principally EPA 
and the National Institutes of Health. Examples of current collaborations between 
industry and governmental agencies include several ongoing projects between the 
Hamner Institute for Health Sciences and the EPA’s National Center for Computa- 
tional Toxicology (NCCT) and its National Center for Environmental Assessment 
(NCEA). 

Other collaborative projects funded by the LRI extend ongoing work at the Na- 
tional Institute of Environmental Health Sciences (NIEHS). The unprecedented col- 
laborations that the LRI has fostered among industry, governmental and regulatory 
agencies, and academia and demonstrates how an industry-sponsored initiative can 
effectively partner with other stakeholders to provide knowledge for science-in- 
formed decisions. 

Among the collaborative research supported by the LRI program: 

• Efforts to deliver state-of-the-art exposure science to advance the 
ExpoCast®component of EPA’s ToxCast®program. 

• Advance the interpretation of high-throughput data. 


estimates that as a whole, the business of chemistry spent some $55 billion on re- 
search and development in 2010, the last year for which complete data are available. Slightly 
over 40% of that amount was spent on basic and applied research. 
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• Accelerate the paradigm shift in chemical risk assessment by incorporating 
ToxCast®data and toxicogenomics information into EPA’s NexGen Risk Assess- 
ment program. 

• Support validation of innovative biomarkers of cumulative exposures. 

• Promote development of alternatives to animal testing. 

The LRI program adheres to a stringent set of principles designed to ensure that 
the collaborative research we fund meets the highest standards for scientific excel- 
lence, transparency, and fairness. 

The LRI program is focused not only on the new technologies for toxicological test- 
ing that are revolutionizing risk-based decision making, but is also helping to de- 
velop innovative biologically relevant approaches to understanding exposure. These 
technologies present an opportunity to develop a new paradigm for toxicity testing 
of chemicals, facilitate understanding of chemical hazards, and improve chemical 
safety evaluations. The current problem that they present is the growing gap be- 
tween the advancements in these new technologies and the science to interpret and 
understand the emerging data. 

In addition to providing state-of-the-art science and technology for chemical safety 
and risk assessments, LRI promotes development of tools that can be used by chem- 
ical companies for product innovation. For example, the LRI currently manages one 
of the most comprehensive portfolios of exposure projects that relates directly into 
efforts to (a) predictively develop exposure information, and (b) make existing expo- 
sure data widely available. Without these tools and data, there would be an in- 
creased likelihood that the next generation of risk assessments would be based en- 
tirely on hazard information or on overly conservative exposure assumptions. 

ACC has suggested to ERA that the transition to new integrative and predictive 
molecular and computational techniques can be enhanced by focusing on critical 
issues such as: 

• The need for an improved understanding of what short-timescale in vitro assays 
can foretell about the likelihood of long-timescale processes that lead to in vivo 
toxicity endpoints, For example, specific response profiles in certain in vitro 
assays or combinations of assays could provide insights into potential toxicity 
endpoints, such as cancer, and may be useful in such decisions as prioritizing 
chemicals for additional testing. Considerable work is underway to enhance con- 
fidence in the use of these approaches and better interpret the results. 

• The value of increased collaboration and engagement across the scientific com- 
munity to interpret ToxCast®data for chemical prioritization. Increased trans- 
parency of relevant data and algorithms will allow ERA to leverage its intellec- 
tual resources and garner stronger understanding of and support for its ap- 
proaches. era’s NexGen Risk Assessment process already provides a similar 
mechanism to engage experts and stakeholders in the emerging science. 

Conclusion 

Ensuring that ERA decision making is firmly based on the use of high-quality 
science is critical to helping the Agency meet its obligation to protect human health 
and the environment. This can be achieved by common-sense reforms that will lead 
to more efficient and effective regulatory decisions. ACC looks forward to working 
with members of the Subcommittee to ensure that the science and processes that 
support the important regulatory work of the Federal Government meet the highest 
standards for quality and reliability. 

Chairman Harris. Thank you very much. 

I now recognize our fourth witness, Dr. Richard Belzer of Regu- 
latory Checkbook. 

STATEMENT OF DR. RICHARD BELZER, 

PRESIDENT, REGULATORY CHECKBOOK 

Dr. Belzer. Chairman Harris, Ranking Member Miller, thank 
you for the invitation to testify. My views are my own and based 
on over 25 years of experience in this field. 


iiJudson, R., et al., In Vitro Screening of Environmental Chemicals for Targeted Testing 
Prioritization: The ToxCast Project, Environmental Health Perspectives 118:485-492 (2010). 
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Previous witnesses last November have testified that EPA’s 
science is not well. The symptoms include the politicization of 
science, which occurs if agency policy officials change scientific in- 
formation to benefit their preferred policy decisions; the 
scientization of policy, which occurs when agency risk assessors 
make policy decisions behind a veil of science; insufficient trans- 
parency reproducibility in agency risk assessments, perhaps best 
exemplified by the National Academy’s recent report on formalde- 
hyde; and dissatisfaction with EPA peer review, perhaps best re- 
vealed when Congress seeks peer review by the academy instead. 

My analysis leads to a four-part diagnosis. First, the 
politicization of science is not something only EPA policy officials 
might do. EPA staff also politicize science, such as when they 
choose a desired result and seek only the science that supports it. 
Second, the scientization of policy is not something only EPA risk 
assessors might do. EPA officials scientize policy, such as when 
they say their policy decisions merely follow the recommendations 
of the scientists. Third, EPA risk assessments are not objective. 
This is not my opinion. An agency staff report says “EPA’s policy 
is that risk assessment should not knowingly underestimate or 
grossly overestimate risks.” Fourth, EPA peer review often does not 
serve the purposes for which it presumably was intended. EPA 
guidelines to the contrary, information quality is irrelevant to 
agency peer review. EPA always controls the charge and often the 
experts, who are often required to interpret science through EPA’s 
policy lenses. Amazingly, peer reviewers often are charged with in- 
directly reviewing their own work. Fifth, EPA advisory committees 
are especially susceptible to politicizing science and scientizing pol- 
icy. In April 2008, CASAC protested then-Administrator Johnson’s 
decision to set the primary National Ambient Air Quality Standard 
for Ozone at .075 parts per million. CASAC members had every 
right to disagree with that decision on policy grounds, but they 
went off the rails, saying that it was their “consensus scientific 
opinion that his decision violated the Clean Air Act.” Science can- 
not determine what is “requisite” to protect public health or what 
constitutes an “ample” margin of safety. Those lie beyond science. 

I recommend several possible remedies for your consideration. 
First, Congress could require the EPA risk assessments, principal 
components, and key studies adhere to information quality prin- 
ciples and standards. Risk assessment should not be based on un- 
disclosed data or models unless perhaps national security is at 
stake. It also should be objective. Deciding how much precaution 
should be accounted for in regulatory decisions is not part of an an- 
alyst’s job. It belongs to the relevant agency official. 

Second, Congress could revamp the EPA’s peer review practices 
to explicitly require them to invest in information quality, to strict- 
ly limit peer reviews to science, and to remove the agency’s ability 
to substantially control outcomes through procedural means. 

Third, Congress could require advisory committees to “establish 
and maintain a clear conceptual distinction between assessment of 
risk and consideration of risk management alternatives” and en- 
sure that their reports “clearly distinguish between the scientific 
basis and the policy basis for the conclusions and recommenda- 
tions.” That advice isn’t mine. It was made by the National Acad- 
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emy committee that wrote the 1983 Red Book. It makes perfect 
sense for advisory committees. EPA officials making decisions 
should not have to struggle to discern where an advisory commit- 
tee’s scientific review ends and its policy advice begins. 

These reforms would go a long way toward improving the quality 
of the EPA science. None of them would politicize science or 
scientize policy. Indeed, they would help prevent both by making 
them stick out like sore thumbs. 

Thank you again for the opportunity to testify. I am happy to an- 
swer any questions when the time permits. 

[The prepared statement of Mr. Belzer follows:] 

Prepared Statement of Dr. Richard Belzer, 

President, Regulatory Checkbook 


Introduction 

Chairman Harris, Ranking Member Miller, and Members of the Subcommittee, 
thank you for inviting me to testify on “Fostering Quality Science at EPA: Perspec- 
tives on Common-Sense Reforms.” I am Dr. Richard B. Belzer, president of Regu- 
latory Checkbook, a nonpartisan, nonprofit organization whose mission includes the 
promotion of quality improvements in science, economics, and information quality. ^ 

I was elected Treasurer of the Society for Risk Analysis in 1998 and 2000, and 
earned its Outstanding Service Award in 2003. Previously I was named a Fellow 
of the Cecil and Ida Green Center for the Study of Science and Society. In 2009 and 
2011, I was elected SecretaryATreasurer of a new professional organization, the Soci- 
ety for Benefit-Cost Analysis. 

From 1988 through 1998, I was a career economist in OMB’s Office of Information 
and Regulatory Affairs, where I reviewed many risk assessments that were integral 
parts of agencies’ Regulatory Impact Analyses, for it is impossible to estimate costs 
and benefits without first estimating risks. My job was to examine agency analyses 
of the risks, costs, and benefits of draft regulations, and present to 0MB officials 
and other Executive Office staff the most objective portrayal possible. Typically, this 
could not be done based on the risk assessments performed by the agencies. Agency 
risk assessments were purposefully biased to make risk appear greater than it was 
and the benefits of regulation appear greater than they were. 

I have been president of Regulatory Checkbook since its founding in 2001. Regu- 
latory Checkbook does not lobby or take public positions on substantive legislation 
or rule making; there is no shortage of organizations committed to doing that. Our 
sparsely populated niche is to seek improvements in the quality of risk assessment 
and economic analysis regardless of whether it tends to support or oppose specific 
regulatory actions. For that reason, we are interested in how quality is affected by 
various procedures, such as public comment, peer review, information quality prin- 
ciples and standards, and Executive oversight. No one has compensated Regulatory 
Checkbook or me for my testimony. 

I am familiar with testimony previously provided to the Subcommittee. I will try 
to build on that and not be redundant. 

Symptoms of the Quality Deficit 

The purpose of these hearings has been to identify ways to improve the quality 
of science used by EPA in regulatory decision making. This, of course, implies that 
the state of the science for science at the Agency is not well. Numerous symptoms 
have been identified. 

Politicization of Science or Scientization of Policy? 

In March 2009, President Obama issued a Memorandum on Scientific Integrity 
stating, among other things, “Political officials should not suppress or alter scientific 
or technological findings and conclusions.”^ The President also made a commitment 
to transparency, saying, “If scientific and technological information is developed and 


1 The views expressed here are my own and do not necessarily represent those of Regulatory 
Checkbook. 

2 Barack Obama. “Scientific Integrity.” Federal Register, 2009, 74(46), 10671-10672. 
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used by the Federal Government, it should ordinarily be made available to the pub- 
lic.” It is my observation, based on over 20 years in risk assessment, that these prin- 
ciples are are universally agreed to — in principle. Putting them into policy turns out 
to be more difficult. It took 22 months for the White House Office of Science and 
Technology Policy — an office whose director the President directly supervises — to 
issue guidance implementing his memorandum. ^ 

Moreover, OSTP’s guidance is crafted with considerable structural and procedural 
ambiguity. 4 It calls for “policymakers [to] involve science and technology experts 
where appropriate,” without clearly stating the circumstances where it wouldn’t be. 
It directs agencies to select candidates for scientific positions “based primarily on 
their scientific and technological knowledge, credentials, experience, and integrity,” 
thereby leaving wide open the option of giving substantial weight to their political 
affiliation or policy views. ® It calls for “independent peer review by qualified ex- 
perts,” but only “where feasible and appropriate.” The guidance says “political offi- 
cials should not suppress or alter scientific or technological findings,” but it does not 
actually generally prohibit this practice. ® Only agency public affairs officers are ex- 
pressly forbidden from doing this. 

The lesson from this is that it is much easier to announce a policy that seems 
straightforward than to implement it. It turns out that the intersection of policy and 
science is a lot more complicated than newspaper reporters, activists, and even can- 
didates for president might think. 

As the Subcommittee has heard, the 2009 report of the Bipartisan Policy Center’s 
Science Policy Project discreetly pointed in a different direction — what is increas- 
ingly being called “the scientization of policy.”® The BPC’s Science Policy Project in- 
cluded former policy officials who, unsurprisingly, had a different perspective on the 
policy/science divide. Former OIRA Administrator Susan Dudley’s testimony to the 
Subcommittee appears to be indicative of this, presumably reflecting her own expe- 
rience. ® 

It turns out that this is an old issue. In 1986, Harvard Kennedy School professors 
Albert Nichols and Richard Zeckhauser published papers claiming that cancer risk 
was systematically overstated at EPA. They wrote that this was done by Agency 
risk assessors for the purpose of influencing risk management decisions. 

In 1990, the Office of Management and Budget elaborated upon this problem in 
its Regulatory Program of the United States Government: 

Unfortunately, risk-assessment practices continue to rely on conservative mod- 
els and assumptions that effectively intermingle important policy judgments 
within the scientific assessment of risk. Policymakers must make decisions 
based on risk assessments in which scientific findings cannot be readily dif- 
ferentiated from embedded policy judgments. This policy environment makes it 
difficult to discern serious hazards from trivial ones, and distorts the ordering 


^John P. Holdren. “Memorandum for the Heads of Executive Departments and Agencies: Sci- 
entific Integrity,” Office of Science and Technoloy Policy, 2010. 

4 OSTP’s guidance is mostly hortatory, saying agencies “should” do various things 19 times 
hut never saying “shall” or “must.” Eight times, these suggestions apply only if the agency 
judges them to be “appropriate.” Four times, they apply only if “practicable.” 

^ President Obama’s memorandum did not include this qualification, stating: “The selection of 
scientists and technology professionals for positions in the executive branch should be based on 
their scientific and technological knowledge, credentials, experience, and integrity.” 

®The President’s memorandum went further, saying “Political officials should not suppress or 
alter scientific or technological findings and conclusions'^ (emphasis added). The difference is 
surely not accidental, but its significance is not transparent. Possibly it shows that the White 
House has learned about the scientization of policy. 

"^Holdren (2010, p. 2). “In no circumstance may public affairs officers ask or direct Federal 
scientists to alter scientific findings.” 

® Bipartisan Policy Center. “Improving the Use of Science in Regulatory Policy,” Washington, 
D.C.: Bipartisan Policy Center, 2009, p. 15. “[SJome disputes over the ‘politicization’ of science 
actually arise over differences about policy choices that science can inform, but not determine.” 

^ Susan E. Dudley. “Written Testimony Before the U.S. House of Representatives Committee 
on Science, Space, and Technology, Subcommittee on Energy and the Environment, Hearing on 
‘Fostering Quality Science at EPA: Perspectives on Common Sense Reforms,’ ” 2011. 

10 Albert L. Nichols and Richard J. Zeckhauser. “The Dangers of Caution: Conservatism in As- 
sessment and the Mismanagement of Risk,” Smith, Advances in Applied Micro-Economics: Risk, 
Uncertainty, and the Valuation of Benefits and Costs. Greenwich, (lonn.: JAI Press, 1986a, 55— 
82. For a less technical version of this paper, see . “The Perils of Prudence: How Con- 

servative Risk Assessments Distort Regulation.” Regulation, 1986b, 10(6), 13—24. 

HR is not clear, however, if this practice illustrates the scientization of policy, or the 
politicization of science by Agency staff rather than by Agency policy officials. It may have ele- 
ments of both. 
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of the Government’s regulatory priorities. In some cases, the distortion of prior- 
ities may actually increase health and safety risks. 

0MB noted with approval the recommendation made by the committee that wrote 
the National Research Council’s 1983 Red Book: 

Regulatory agencies should take steps to establish and maintain a clear concep- 
tual distinction between assessment of risks and the consideration of risk man- 
agement alternatives; that is, the scientific findings and policy judgments em- 
bodied in risk assessments should be explicitly distinguished from the political, 
economic, and technical considerations that influence the design and choice of 
regulatory strategies. 

Like the authors of the Red Book, 0MB thought that fidelity to the Red Book rec- 
ommendations was at least part of the solution. Though it was published more than 
20 years ago, the problem 0MB highlighted is the same thing that the BPC identi- 
fied. 

While this is an old story, that does not mean it is outdated. In 2004, ERA pub- 
lished a staff report that explains its risk assessment policies and practices with sig- 
nal clarity. This report acknowledges that ERA risk assessments are intentionally 
biased to overstate risk, and that this is done for the purpose of scientizing policy: 

[SJince ERA is a health and environmental protective agency, ERA’s policy is 
that risk assessments should not knowingly underestimate or grossly overesti- 
mate risks. 

All risk assessments err because they are estimates. What the ERA staff said is 
that they have a strong preference for erring on the side of overestimating risk, just 
not “grossly” overestimating it. They justify this preference based on the “health and 
environmental mission” of the Agency. This preference for overestimating the mag- 
nitude of risk is in addition to a preference for erring on the side of promulgating 
regulations that err on the side of overprotection. 

It is worth reflecting on what this would mean if other federal agencies did the 
same thing: 

• Would it be reasonable for engineers at the Eederal Aviation Administration to 
intentionally overestimate the risk of air travel, perhaps by assuming all air- 
craft were as risky as the riskiest of them, and use those overestimates to moti- 
vate the FAA Administrator to promulgate more stringent safety regulations for 
all aircraft? 

• Would it be reasonable for examiners in the Department of the Treasury to 
knowingly overstate the risk that a major bank might fail, in order to persuade 
the Secretary to take over that bank? 

• Would it be reasonable for analysts at the Central Intelligence Agency to pur- 
posefully overstate the likelihood that the Islamic Republic of Iran will succeed 
in developing an fielding a nuclear weapon, thereby encouraging the Rresident 
to launch a preemptive military attack? 

To ask these questions is to answer them. It is the obligation of federal risk asses- 
sors, no matter where they work, to estimate risk as objectively as possible. They 
should never misuse the tools of risk assessment to manipulate decision makers into 
taking specific actions. Remarkably, the ERA staff report denies that the discretion 


Office of Management and Budget. “Current Regulatory Issues in Risk Assessment and Risk 
Management,” Regulatory Program of the United States, April 1, 1990— March 31, 1991. Wash- 
ington, DC: Office of Management and Budget, 1990, 13—26. Full disclosure: I was the author 
of OMB’s white paper. It is out of print hut available on my Web site athttp:! / 
www.rbhelzer.com ! uploads 17111714171 74353 /omb -1990-risk-assessment.pdf. 

12 National Research Council. Risk Assessment in the Federal Government: Managing the 
Process. Washington, D.C.: National Academies Press, 1983, p. 151. 

11 Bipartisan Policy Center (2009, p. 13). “Political decision makers should never dictate what 
scientific studies should conclude, and they should base policy on a thorough review of all rel- 
evant research and the provisions of the relevant statutes. But some disputes over the 
‘politicization’ of science actually arise over differences about policy choices that science can in- 
form, but not determine” (p. 13). 

i®U.S. Environmental Protection Agency Office of the Science Advisor. “An Examination of 
EPA Risk Assessment Principles and Practices; Staff Paper, EPA/lOO/B-04/001,” 2004, p. 13. 
This does not necessarily mean that EPA always succeeds in overstating risk, or that there are 
not circumstances in which EPA does not understate risks, whether by accident or intent. 
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of Agency policy officials is constrained or misdirected by their practice of purpose- 
fully overestimating risk, 

Diagnosis 

Whether science has been politicized or policy has been scientized is a useful dis- 
tinction, but it is not complete. For example, it is assumed that science is politicized 
when policy officials invade the space of the scientists; and conversely, policy is 
scientized when agency scientists attempt to make policy decisions that Congress 
has delegated to agency heads. 

This model is incomplete because Agency policy officials and risk assessors appear 
equally prone to do both. Sometimes, it is agency policy officials who scientize policy, 
such as when they try to attribute their policy choices to science. A policy official 
can avoid a lot of controversy if he is perceived as “merely following the science.” 

Agency risk assessors may be willing or even pleased to go along, for it increases 
their power and authority inside the agency, in its battles with 0MB, and for de- 
flecting Congressional criticism. Thus, Agency risk assessors may have no more in- 
terest than Agency policy officials in revealing the extent to which officials have at- 
tributed policy decisions to science. Similarly, Agency officials and risk assessors 
alike may prefer not to make transparent the extent to which risk assessors actually 
make policy decisions under the cover of science. 

Conflict arises, however, when Agency officials and risk assessors do not agree on 
policy. In these situations. Agency policy officials must first reclaim from Agency 
risk assessors the authority delegated to them by Congress to make policy decisions. 
It is easy for risk assessors to accuse their political bosses of politicizing science and 
nearly impossible for policy officials to defend themselves when the charge is false. 

On the other hand, sometimes it is Agency risk assessors who politicize science. 
This happens when risk assessors choose the best available science that supports 
their preferred policy decision. Few policy officials would ever be the wiser, because 
it requires from them independent scientific expertise, substantial issue-specific 
knowledge, and more time than they have available. 

The desired principles can be clearly expressed, if not easily implemented: 

• Agency policy officials should be limited to making policy. 

• Agency risk assessors should be limited to assessing risk. 

• Risk assessment should be performed as objectively as possible and not be mis- 
used as a tool for achieving policy objectives through the back door. 

Policy officials should stay out of science. They should allow science to inform 
their decisions but never allow it to control them, never hide behind it, and never 
tell scientists what conclusions to reach. They also should be persistent about ask- 
ing risk assessors the right questions and getting second opinions from external au- 
thorities. 

This goal begins with the Red Book recommendation and goes much further. 
Whereas the Red Book authors envisioned a smoothly interactive and iterative rela- 
tionship between risk assessors and risk managers, with a “clear conceptual distinc- 
tion” between science and policy “established and maintained,” 30 years of history 
has shown that this model has either failed or cannot be implemented in a real- 
world regulatory agency, 


i®U.S. Environmental Protection Agency Office of the Science Advisor (2004, pp. 14-16). 

I'^In previous testimony to this Subcommittee, EPA Administrator Jackson’s decision to revise 
the 2008 National Ambient Air Quality Standard for ozone following the recommendations of 
the Clean Air Scientific Advisory Committee was described by a former CASAC chairman in 
similar but more strident terms. See Roger 0. McClellan. “Written Testimony Before the U.S. 
House of Representatives, Committee on Science, Space, and Technology, Subcommittee on En- 
ergy and the Environment, Hearing on 'Quality Science for Quality Air,’ ” 2011. 

i®The author of this recommendation on behalf of the Red Book committee believes that EPA 
officials misinterpreted and misapplied it. See D. Warner North. “Reflections on the Red/Mis- 
Read Book, 20 Years After.” Jourtial of Human and Ecological Risk Assessment , 2003, 9(5), 
1145—1154. A somewhat different interpretation is that implementation as envisioned by the 
Committee was not administratively or politically feasible, an interpretation Professor Merchant 
appears to favor. See Gary E. Merchant. “Written Testimony Before the U.S. House of Rep- 
resentatives Committee on Science, Space, and Technology, Subcommittee on Energy and the 
Environment, Hearing on ‘Fostering Quality Science at EPA: Perspectives on Common Sense Re- 
form,’ ” 2011, p. 5. “As the role of science becomes ever more important to EPA’s mission, and 
as the perception of EPA’s science continues to be skeptical across the political spectrum, it may 
be time to consider a different model that institutionally separates the generation and assess- 
ment of science from the application of that science in regulatory decision making.” Marchant 

Continued 
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Information Quality Principles and Standards 

In the following sections I focus on three areas in which EPA science has specific, 
notable deficiencies. These are information quality, peer review, and the confused 
role of federally chartered advisory groups. 

In 2002, 0MB issued governmentwide guidelines to implement a statutory di- 
rective to improve information quality. Like almost every other covered agency, 
EPA issued its own agency-specific guidelines before the October 1, 2002, dead- 
line. These guidelines commit EPA to adhere to certain standards of trans- 
parency, reproducibility, integrity, objectivity, and utility, and to establish adminis- 
trative mechanisms whereby any person may seek and obtain the correction of non- 
compliant information. Indeed, EPA expressed the view that adhering to OMB’s 
guidelines would not pose any challenge because its existing policies and procedures 
already ensured and meiximized information quality. 

EPA’s information quality guidelines say the Agency “is dedicated to the collec- 
tion, generation, and dissemination of high quality information” and “seeks to foster 
the continuous improvement of existing information quality activities and pro- 
grams.” “In implementing these guidelines,” EPA said “ensuring the quality of infor- 
mation is a key objective alongside other EPA objectives, such as ensuring the suc- 
cess of Agency missions, observing budget and resource priorities and restraints, 
and providing useful information to the public.” EPA also established well-defined 
administrative procedures for managing requests for correction and administrative 
appeals. 

To be clear, information quality standards are expansive. They apply to “any com- 
munication or representation of knowledge such as facts or data, in any medium or 
form” — but not to policy decisions. Thus, they apply to risk assessment documents 
to the extent that they contain “representationfs] of knowledge such as facts or 
data.” Because EPA officials claim that Agency risk assessment products are sci- 
entific, there is no doubt that they are fully covered by applicable information 
quality guidelines. 

The Subcommittee should be aware that EPA has exempted press releases and 
fact sheets from its information quality guidelines, which it describes as 
“[ijnformation of an ephemeral nature.” Given that press releases and fact sheets 
are often the only information Congress and the press know about a complex risk 
issue, this exemption is obviously problematic. Further, the Subcommittee should be 
aware that EPA also exempts “[ijnformation presented to Congress as part of the 
legislative or oversight processes.” EPA testimony may have many desirable at- 
tributes, but adherence to information quality principles and standards is not one 
of them. 

Many error correction requests submitted to EPA concern Agency risk assess- 
ments or components thereof, which the petitioner claims contain factual errors. 
Some requests are intended to seek full disclosure of data and methods to enable 
third parties to test for error, which both OMB’s and EPA’s information quality 
guidelines require. 

EPA committed to respond to requests for correction and appeals within 90 days. 
EPA’s actual performance, however, has not lived up to these commitments. As of 
September 30, 2010, EPA’s average response time for a request for correction was 
no less than 166 days. EPA’s average response time for an appeal was no less than 


does not credibly explain how an external science production entity, such as his proposed Insti- 
tute for Scientific Assessments, staffed and managed by full-time federal employees, would not 
succumb to the twin temptations of politicization and scientization. 

Office of Management and Budget. “Guidelines for Ensuring and Maximizing the Quality, 
Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies; Notice; Re- 
publication.” Federal Register, 2002, 67(36), 8452-8460. 

“Information Quality Act.” 44 U.S.C. 3516 note. 2000. 

21 U.S. Environmental Protection Agency. “Guidelines for Ensuring and Maximizing the Qual- 
ity, Objectivity, Utility, and Integrity of Information Disseminated by the Environmental Protec- 
tion Agency (EPA/260R-02-008),” 2002. 

^^U.S. Environmental Protection Agency (2002, pp. 10-14). 

2®U.S. Environmental Protection Agency (2002, p. 10). 

24 Paul Anastas. “Written Testimony Before the U.S. House of Representatives Committee on 
Science Space and Technology, Subcommittee on Oversight, Hearing on ’EPA’s Integrated Risk 
Information System,’ ” 2011, p. 1. “IRIS assessments provide a scientific foundation for EPA de- 
cisions to protect public health across EPA’s programs and regions under an array of environ- 
mental laws” (emphasis added). 

25 U.S. Environmental Protection Agency (2002, pp. 16-17). 

25 Some requests for correction are misguided attempts to change regulatory decisions. How- 
ever, these requests are easy for EPA to dismiss on the ground that they concern matters that 
are exempt from the information quality paradigm. 
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316 days.^'^ These figures are biased downward, and thus understate EPA’s dilatory 
behavior, because they include requests and appeals that were still open at the end 
of FY 2010. 28 

In short, EPA’s well-written administrative procedures have in practice failed to 
enable affected persons to “seek and obtain” the correction of information that does 
not comply with applicable information quality principles. Assuming it takes only 
46 days to review EPA’s response to a request for correction and file an appeal, it 
has taken on average more than 527 days for EPA’s internal administrative process 
to run its course. 

The substantive merits of these requests for correction vary, but it cannot be de- 
nied that many are highly meritorious. This can be seen by reviewing specific re- 
quests or logically inferred by the length of time EPA takes to respond. It should 
be easy for the Agency to quickly refute requests for correction that lack any merit, 
especially those which impermissibly seek to challenge Agency policy decisions. Con- 
versely, requests for correction that are highly meritorious could be very hard to re- 
fute. If acknowledging error would undermine the legal standing or political legit- 
imacy of a major regulation or an important EPA policy, no one should be surprised 
that the Agency takes a long time to decide how to respond, or that its responses 
are ambiguous, technically weak, misleading, or flatly wrong. 

If an agency’s response to a request for correction is incomplete, misguided, or 
lacks merit, the only recourse is an appeal within the agency. One cannot appeal 
to another Executive branch agency or seek review by a federal court. For that rea- 
son, public enthusiasm is limited even for submitting the most meritorious of error 
correction requests. Governmentwide, the number of requests for correction filed an- 
nually has declined by more than 75% since FY 2003. This is not because federal 
agencies have suddenly stopped disseminating erroneous information. It is because 
the agencies have responded to the Information Quality Act as if it were a poten- 
tially lethal virus and developed effective antibodies to prevent reinfection. 

Peer Review 

EPA is perhaps the federal agency that has committed the most to peer review. 
It conducts numerous peer reviews every year and has published a series of hand- 
books that guide Agency staff through the process, Nevertheless, there appears 
to be widespread dissatisfaction with the actual performance of EPA peer review. 
This is self-evident given Congress’ repeated decisions to supplement or even bypass 
EPA peer review in favor of the National Academy of Sciences. 

Several problems afflicting EPA’s peer review program are discussed below. 

OMB’s Bulletin on Peer Review Contains Obvious Errors 

0MB issued governmentwide guidance on peer review in 2005. This guidance 
is generally very useful and helpful. For example, it clearly states, “Peer reviewers 
shall be charged with reviewing scientific and technical matters, leaving policy de- 
terminations for the agency.” 

But OMB’s guidance is especially weak exactly where it should have been strong- 
est. Even though enhancing information quality was its raison d’etre, the guidance 
includes no requirement that agencies actually make information quality principles 
and standards an integral part of scientific peer review. 0MB waffles, sa 3 dng 
“[rleviewers shall be informed of applicable access, objectivity, reproducibility and 
other quality standards under the Federal laws governing information access and 
quality.” 21 In short, an EPA peer review complies with OMB’s guidance as long as 
peer reviewers are informed about information quality, perhaps similar to one of the 
dozens of disclosure forms that must be provided at settlement when purchasing a 


2iRichard B. Belzer. “Risk Assessment and Information Quality: An Empirical Study of Fed- 
eral Agency Performance, 2010 Update,” Society for Risk Analysis 2010 Annual Meeting, Salt 
Lake City, Ut., 2010. Since this paper was presented, EPA has received four new requests for 
correction. 

28 Eleven of 44 requests for correction and one of 16 appeals remained open at the end of FY 
2010. The average response time, once these open actions were resolved, could only be greater. 

29U.S. Environmental Protection Agency. “Peer Review Handbook (1st Ed.),” Washington, 
D.C.: U.S. Environmental Protection Agency Science Policy Council, 1988, . “Peer Re- 

view Handbook (2d Ed.),” Washington, D.C.: U.S. Environmental Protection Agency Science Pol- 
icy Council, 2000, . “Peer Review Handbook (3rd Ed.),” Washington, D.C.: U.S. Envi- 

ronmental Protection Agency Science Policy Council, 2006. 

88 Office of Management and Budget. “Final Information Quality Bulletin for Peer Review.” 
Federal Register, 2005, 70(10), 2664-2667. 

8:^ Office of Management and Budget (2005, p. 2675). Emphasis added. 
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house. There is no obligation for peer reviewers to do anything with this informa- 
tion. 

OMB’s guidance also includes a pair of extraordinarily large loopholes. First, 
0MB allows agencies to infer that studies published in peer-reviewed literature ad- 
here to information quality principles and standards, including the crucial stand- 
ards of presentational and substantive objectivity. This is bizarre. Adherence to 
these principles plays no role in journal review. If they knew about them, some edi- 
tors of scholarly journals probably would consider information quality principles and 
standards wholly irrelevant or contradictory to the journal’s mission. No matter; to 
0MB, peer review by a scholarly journal means the information contained in it is 
presentationally and substantively objective. 

Second, 0MB exempts reports of the National Academy of Sciences from any scru- 
tiny whatsoever. This is true even if there is no evidence that the review took ac- 
count of applicable information quality principles and standards or there is incon- 
trovertible evidence that the review violated these principles and standards. 

Information Quality Is AWOL from EPA Peer Review 

EPA’s latest Peer Review Handbook mentions information quality several places, 
but each reference is little more than boilerplate. Here is the most substantive ref- 
erence I can find: 

The Agency recognizes peer review as a component of pre-dissemination review 
that complements and enhances the “objectivity” and “utility” of EPA’s informa- 
tion products. The Agency recommends that offices conduct pre-dissemination 
reviews of information to ensure that the information is of appropriate quality 
before it is disseminated to the public. Pre-dissemination review is especially 
important for influential scientific information and highly influential scientihc 
assessments. 

Notice that pre-dissemination review, which applicable information quality guide- 
lines require agencies to perform, is reduced to a mere recommendation. The Hand- 
book does not even include OMB’s requirement that peer reviewers be “informed” 
about information quality principles and practices, so it should surprise no one 
when they aren’t. 

A reasonable inference is that EPA’s Science Policy Council, the author of the 
Peer Review Handbook, does not want information quality to play a meaningful role 
in Agency peer review. Rather, the SPC hopes that by conducting peer review EPA 
will be treated as if it had complied with information quality principles and stand- 
ards. This is wholly unjustihed. Peer reviews conducted fully in compliance with the 
letter of the Handbook do not and cannot adhere to information quality principles 
and standards because those principles and standards are AWOL. 

EPA’s Science Advisory Board and the National Academy of Sciences are not solu- 
tions to this problem, for their reviews are no more likely to take information qual- 
ity seriously. To take one obvious example, many observers have strongly endorsed 
Chapter 7 of the Academy’s review of EPA’s draft assessment of formaldehyde as 
a highly desirable step forward for improving the quality of IRIS assessments. 
Perhaps it is, but the Academy’s formaldehyde report does not include adherence 
to information quality principles and standards anywhere in its “road map.” Indeed, 
the report never even mentions information quality, which suggests that the form- 
aldehyde committee was utterly unaware of EPA’s information quality guidelines. 

For this reason, the Chapter 7 “road map” might not be as helpful as its advocates 
hope. Most disturbingly, any Congressional directive to EPA insisting that it adhere 


Office of Management and Budget (2005, p. 2675). “Principal findings, conclusions and rec- 
ommendations in official reports of the National Academy of Sciences are generally presumed 
to have been adequately peer reviewed.” 0MB may have tried to hedge this blanket endorse- 
ment by limiting it to “principal findings,” but the effectiveness of this hedge seems likely to 
be ephemeral. 

33 An incontrovertible violation would occur in any instance where the Academy gives policy 
advice. See the discussion surrounding footnote 30. 

3‘‘U.S. Environmental Protection Agency (2006, p. 17). 

33 National Research Council. Review of the Environmental Protection Agency’s Draft IRIS As- 
sessment of Formaldehyde. Washington, D.C.: National Academies Press, 2011. 

33 This does not mean the formaldehyde committee ignored information quality in its review. 
Several places in the report one can find discussions that indicate the committee wrestled with 
quality issues. Similarly, the “road map” has numerous references to “quality” because it wanted 
EPA to focus on “high-quality” studies. But the committee was bereft of a framework for defin- 
ing quality because it apparently knew nothing about EPA’s Information Quality Guidelines. 
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to the “road map” is an implicit invitation for the Agency to ignore information qual- 
ity. Surely Congress did not intend this to happen. 

Noncompliance with the Peer Review Handbook 

There are numerous anecdotes suggesting that EPA peer reviews do not actually 
comply with the Peer Review Handbook. I am unaware of any systematic research 
on this question that would permit a more comprehensive inference. Clearly, such 
research could be valuable if it were conducted rigorously and independently of 
EPA. 

Obviously, if research showed that EPA’s adherence the Peer Review Handbook 
was as spotty as is the Agency’s adherence to its information quality guidelines, this 
might go a long way toward explaining why there appears to be such widespread 
dissatisfaction with EPA peer review. Research also could discover if noncompliance 
with the handbook was random or causally associated with specific issues or regu- 
latory programs. 

Exeessive Ageney Control 

EPA’s Peer Review Handbook makes clear that the Agency retains full control 
over the peer review charge and de facto control over the selection of peer reviewers. 
This is obviously true when a peer review is conducted by a panel established by 
EPA under the Federal Advisory Committee Act (FACA). But it is also true when 
the EPA conducts a workshop or contracts with a private company to conduct peer 
review. In these circumstances, EPA still controls the charge^® and has the author- 
ity to veto the contractor’s selection of peer reviewers. 

Nonscientific Content in EPA’s Charge to Scientific Peer Reviewers 

EPA peer review panels often are given a charge that includes crucial nonsci- 
entific content for which scientists have no special skills or insights. This occurs, 
for example, when a peer review panel is constrained to look at scientific informa- 
tion through the Agency’s policy lenses. Common examples include the derivation 
of unit risk factors for carcinogens and Reference Doses for noncarcinogens, both of 
which have scientific content but are controlled by policy choices. When a scientific 
peer review panel is asked to review a proposed unit risk factor or Reference Dose, 
it is being asked to ratify the Agency’s policy choices. 

Insufficient Expertise 

By virtue of their size, peer review panels may appear to be capable of reviewing 
all the relevant scientific questions posed by a draft risk assessment. This may not 
be true, however, if the issues presented are very broad and cross multiple dis- 
ciplines. On a panel containing the 15 best external scientists, there may be just 
a couple who have crucial expertise related to a specific issue. If the number of sci- 
entific issues is large, reviewers will be assigned to those issues on which then have 
the most expertise. When it comes time to put the review together, panel members 
will be inclined to jealously guard the portion of the review they performed but defer 
completely to other members with respect to the rest. Instead of a single peer review 
performed by a panel of 15, the final report may be a half-dozen or more separate 
reviews, each performed by a small number of scientists, then repackaged as is it 
were a single document. 

Exeessive Expertise, of a Certain Form 

It is becoming increasingly common to observe a peer review panel consisting of 
experts who are the authors of the research on which EPA has based its risk assess- 
ment. These experts are valuable and important, for they alone can ensure that the 
Agency has interpreted their work correctly. But they have no business serving on 
a peer review panel whose job will be to review whether these studies were per- 
formed correctly, whether they are the best available, whether they are objective, 
etc. 


is an open question whether this question is researchable. “Compliance” with a complex 
guidance document is not a binary state. The research task would involve a painstaking review 
of a representative sample of peer reviews. The sample would have to be large enough to have 
the statistical power to reject the null hypothesis. 

38U.S. Environmental Protection Agency (2006, p. 59). 

®®U.S. Environmental Protection Agency (2006, p. 61). 
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This practice is disturbingly commonplace. The Clean Air Scientific Advisory 
Committee (CASAC), which performs a peer review function under Section 109(d)(2) 
of the Clean Air Act, is chaired by an author of studies on which EPA bases risk 
assessments for mortality caused by ambient air pollutants. Four of the seven cur- 
rent members of CASAC have published research referenced in EPA’s latest Inte- 
grated Science Assessment for ozone, which CASAC is responsible for reviewing. A 
scientist who formerly served on CASAC has testified before this Subcommittee that 
he was also a contributing author of multiple ISAs. It is simply impossible for 
CASAC to independently peer review EPA risk assessment documents that rely on 
its members’ own research. In fact, it violates EPA’s Peer Review Handbook, for it 
represents the ultimate conflict of interest. 

Conflicts of Interest 

Most observers seem to agree that conflicts of interest ought to be avoided if at 
all possible, and that peer review panels should manage bias by ensuring that a 
“balance of biases” is obtained. This principle is key to the National Academy’s 
model, for example. 

I unapologetically take a different view. We are saddled with conflict-of-interest 
policies that were written by lawyers in a way that makes them easy for lawyers 
to implement. They treat appearances the same as facts, and minor financial in- 
terests related to for-profit employment more gravely than huge financial interests 
related to dependence on government research grants. Conflict-of-interest policies 
include measures to balance bias because scientific peer review panels routinely do 
more than review science — they opine on policy. 

Public Participation Is Limited and Public Comments Are Ignored 

EPA’s Peer Review Handbook purports to welcome public participation in peer re- 
view, but it treats the public as a burden to be endured rather than a source of in- 
sight. Similarly, the Handbook endorses the practice of making public comments 
available to peer reviewers, but it does nothing to encourage, never mind require, 
that peer reviewers consider even the most significant scientific content of public 
comments. Unsurprisingly, public comments are routinely ignored in practice, and 
public participation is typically constrained to presentations lasting a few min- 
utes. 


Bell, F. Dominici and J.M. Samet. “A meta-analysis of time-series studies of ozone and 
mortality with comparison to the national morbidity, mortality, and air pollution study.” Epide- 
miology, 2005, 16(4), Michelle L. Bell, Aidan McDermott, Scott L. Zeger, Jonathan M. Samet 
and Francesca Dominici. “Ozone and Short-term Mortality in 95 US Urban Communities, 1987— 
2000.” JAMA, 2004, 292(19), 2372-2378. 

George D. Thurston. “Written Testimony Before the U.S. House of Representatives, Com- 
mittee on Science, Space, and Technology, Subcommittee on Energy and the Environment, Hear- 
ing on ‘Quality Science for Quality Air’: RE: The Science of Air Pollution Health Effects and 
The Role of CASAC in EPA Standard Setting,” 2011, p. 2. 

42 U.S. Environmental Protection Agency (2006, p. 37). “Since it would probably result in a 
perceived, if not real, conflict of interest, the group that is generating the work product usually 
cannot conduct or perform the peer review of its own work product.” 

43 The National Academies. “Policy on Committee Composition and Balance and Conflict of In- 
terest,” The National Academies, 2003. 

44 A more extensive discussion of the contrasts between scholarly and governmental peer re- 
view can be found in a paper I wrote for a 2002 conference sponsored by the Society for Risk 
Analysis. See Richard B. Belzer. “Interests and Incentives in Government Peer Review,” Con- 
flict, Consensus, and Credibility: A Forum on Regulatory Peer Review, Alexandria, VA, 2002. 
Available at http:! lwwiv.rbbelzer.comlpresentations.htmW2002. 

45 Andrew Stark. Conflict of Interest in American Life. Cambridge, Mass.: Harvard University 
Press, 2000. 

46 U.S. Environmental Protection Agency (2006, p. 49). “To ensure that public participation 
does not unduly delay activities. Offices should specify time limits for public participation 
throughout the peer review process.” 

4'7U.S. Environmental Protection Agency (2006, p. 74 [distribution to peer reviewers is re- 
quired for “influential scientific assessments”]). See also p. 49: “When employing a public com- 
ment process as part of the peer review. Offices should, whenever practical, provide peer review- 
ers with access to public comments that address significant scientific or technical issues.” 

43 Robert F. Phalen. “Written Testimony Before the U.S. House of Representatives, Committee 
on Science, Space, and Technology, Subcommittee on Energy and the Environment, Hearing on 
‘Quality Science for Quality Air’: The CASAC— PM Committee — Setting Air Quality Standards,” 
2011. “The public comments were not weighed and discussed by CASAC-PM in spite of the fact 
that most were well-reasoned and relevant. If the agenda included time for discussion of public 
comments and formal acceptance or rejection of their recommendations, the process might be 
improved.” 
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This means peer reviews of draft EPA risk assessments tend to be dialogues be- 
tween the peer review panel and Agency staff, who might (or might not) have writ- 
ten (part of) the document. Unless they happen to be members of the peer review 
panel, primary researchers are rarely present and would in any case be relegated 
to cameo presentations during the limited time permitted for public comment. 

In the section below on remedies, I describe an alternative to this zoological style 
peer review in which public participation is taken seriously, and primary research- 
ers have the lead in presenting scientific information but do not play a role in evalu- 
ating it. 

Federally Chartered Advisory Committees 

Even more than peer review panels, advisory committees are susceptible to politi- 
cizing science and scientizing policy. To the extent that they can locate a scientific 
rationale for the advice they want to provide, it can only make their recommenda- 
tions more persuasive. Like Congress, the public often fondly hopes for scientific an- 
swers to difficult policy questions. If a policy choice can be made to appear scientific, 
it may have a much easier time gaining public acceptance. 

One of the most striking examples of scientization occurred in 2008, after EPA 
finalized its revision to the ozone National Ambient Air Quality Standard. CASAC 
sent Administrator Stephen Johnson an unsolicited letter strenuously disagreeing 
with his decision. By itself, this might have been noteworthy but it should not have 
been overly controversial. After all, advisory committees that are independent of an 
agency’s control must be free to offer whatever policy advice they see fit, and Agency 
officials are never obligated to accept policy recommendations from advisory commit- 
tees. 

But CASAC went much, much further. CASAC misrepresented its policy advice 
as science: 

It is the Committee’s consensus scientific opinion that your decision to set the 
primary ozone standard above this range [0.060 to 0.070 ppm] fails to satisfy 
the explicit stipulations of the Clean Air Act that you ensure an adequate mar- 
gin ofsafety for all individuals, including sensitive populations. 

This is wrong in multiple ways, and it should have drawn widespread opprobrium 
instead of acclaim. Science might be able to determine what human health effects 
occur at defined ozone concentrations, though even this ability becomes suspect as 
concentrations approach background. But it is impossible for science to determine 
what concentration is “requisite to protect the public health” or determine what con- 
stitutes “an adequate margin of safety.” “Requisite” and “adequate” are squishy pol- 
icy terms; they cannot be defined scientifically. But CASAC attempted to scientize 
air pollution policy — to make it appear as if science is the rightful venue for deter- 
mining the meaning of “requisite” and “adequate.” Equally disturbing, CASAC at- 
tempted to arrogate the authority to make these policy decisions despite knowing 
full well that Congress delegated them to the Administrator. 

This incident exposed a serious defect in the Clean Air Act’s procedures, one that 
has lessons for advisory committees generally. By asking CASAC to review the sci- 
entific record to ensure that it “accurately reflects the latest scientific knowledge,” 
but simultaneously ask CASAC to give policy advice to the Administrator con- 
cerning what the standard ought to be. Congress practically invited CASAC to 
scientize policy. For CASAC members, it was their scientific credentials and exper- 
tise that gave them power, which they willfully abused. And because they did so, 
it is entirely reasonable to be skeptical about the quality of CASAC’s scientific re- 
view. Did CASAC also politicize the science to make it support members’ personal 
opinions about air pollution policy? Has anyone conducted a rigorous review to find 
out? 


"^^See, e.g., William C. Adams. “Public Comment to CASAC Ozone Review Panel Teleconfer- 
ence.” Available at http:! /www.epa.gov/sablpdflpub-CommentS-03-05- 

07-dr-wm-adams-UC-davis.pdf; accessed January 29, 2012. 

^*^1 use the term zoological to describe EPA peer reviews to reflect the fact that the public’s 
role is strictly observational. Even tapping on the glass is prohibited. 

^^Rogene Henderson. “April 7, 2008, Letter to Stephen L. Johnson from CASAC on 'Clean Air 
Scientific Advisory Committee Recommendations Concerning the Final Rule for the National 
Ambient Air Quality Standards for Ozone,’ ” CASAC April 7, 2008, Letter on O 2 NAAQS. Wash- 
ington, D.C.: U.S. Environmental Protection Agency Office of the Science Advisory Board, 2008, 
p. 2. Emphasis added. 

See, e.g., “Clean Air Act.” 44 U.S.C. 7401 et seq. 1970. See § 7409(d)(2)(B), referring back 
to 1108(a)(2). 



108 


Some Possible Remedies 

Several remedies can be envisioned that follow from my diagnosis. 

Information Quality 

The key problem noted above is that EPA does not adhere to its information qual- 
ity guidelines. It largely ignores its procedural and substantive commitments. It 
does not respond in a timely manner to requests for correction and appeals. When 
it does respond, it tends to obfuscate. When it acknowledges errors, it does not cor- 
rect them. 

These deficiencies are no doubt caught up in program offices’ desire to defend 
their past or pending regulatory decisions. But that cannot explain the Agency’s un- 
willingness to adhere to information quality principles and standards in its science 
program, which EPA leadership claims is not regulatory. 

A reasonable inference is that EPA’s research programs may be infected by both 
scientization (the desire to make policy decisions through science) and politicization 
(the abuse of science for policy purposes). Requiring EPA research programs to fully 
adhere to information quality principles and standards would go a long way toward 
overcoming these problems if they exist. If they do not exist, then full adherence 
to information quality principles and standards would earn EPA the credibility it 
believes it is deserved, and once and for all refute its many critics. 

There are simple reforms that Congress could make that would breathe life into 
the information quality paradigm, thereby achieving a dramatic improvement in the 
quality of EPA science. In particular, Congress could require one or more of the fol- 
lowing: 

Require Full Disclosure of All Data, Models and Methods for Any Study Used as the 
Basis for a Risk Assessment or Component Thereof 

There appears to be a broad consensus in favor of transparency and reproduc- 
ibility, the two procedural information quality standards. Under applicable informa- 
tion quality guidelines, data, models, and methods must be fully disclosed such that 
qualified third parties can reproduce the agency’s results and obtain essentially the 
same result. If third parties are unable to even make such an attempt, then the 
agency work product is per se insufficiently transparent and violates applicable 
standards. If third parties can make the attempt but cannot reproduce EPA’s re- 
sults, then the information should be presumed to fail the objectivity test. In either 
case, the information involved should not be disseminated, much less used for risk 
assessment. 

Agencies avoid the full force of this transparency standard by claiming, correctly, 
that published articles in scholarly journals do not disclose enough information to 
meet the transparency and reproducibility standards. Congress can best solve this 
problem by altering incentives. 

Contracting regulations already permit federal agencies to demand that recipients 
of federal research funds submit their data upon request. Unfortunately, agencies 
still have the discretion not to ask, and they often do so precisely to avoid having 
to disclose the information to the public as the Shelby Amendment otherwise re- 
quires. Congress could relieve federal agencies of this conundrum by requiring 


^3 This principle is highlighted in previous testimony to the Subcommittee without reference 
to applicable information quality guidelines. See Anastas (2011, p. 1). “IRIS assessments provide 
a scientific foundation for EPA decisions to protect public health across EPA’s programs and re- 
gions under an array of environmental laws. While not regulations, IRIS assessments are crit- 
ical to many Agency decisions. After becoming Administrator in early 2009, Administrator Jack- 
son reviewed the IRIS program and asked the Office of Research and Development (ORD) in 
May 2009 to implement a new IRIS process that would revitalize the program and make it more 
responsive to the needs of the Agency. The aim of the new process was to ensure the highest 
level of scientific quality, integrity, transparency, and timeliness.’’ 

^^The scientific information classification scheme recommended to the Subcommittee by Dr. 
Moghissi also has significant merit as a way to identify where scientific knowledge is weakest 
so that investments in research could be targeted to have the greatest value. See A. Alan 
Moghissi. “Written Testimony Before the U.S. House of Representatives, Committee on Science, 
Space, and Technology, Subcommittee on Energy and Environment, Hearing on ‘Fostering Qual- 
ity Science at EPA: Perspectives on Common-Sense Reform:’ The Need for Regulatory Science 
Transparency at the EPA,” 2011. 

^^Pub. L. 105-277, 112 Stat. 2681-495: “That the Director of 0MB amends Section .36 

of OMB Circular A- 110 to require Federal awarding agencies to ensure that all data produced 
under an award will be made available to the public through the procedures established under 
the Freedom of Information Act.” OMB’s implementation of this provision was highly controver- 
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them to obtain research data if they want to use a federally funded study as the 
basis for risk assessment. Requiring disclosure imposes only trivial costs on the 
agencies and does not violate the contractual terms of any federally funded re- 
searcher. No burden would be imposed on anyone if the agency did not want to use 
a federallyfunded study as the basis for risk assessment, and no researcher would 
be compelled to accept federal research funds to conduct a study likely to be useful 
in risk assessment. 

If an agency wants to rely on a study that was funded by another party, whether 
that be a State, business, trade association, or nongovernmental organization, noth- 
ing currently prevents the agency from asking that this information be supplied, nor 
is there any general legal barrier to the other party providing it. States, businesses, 
trade associations, and nongovernmental organizations that want their research to 
be used for public policy should happily volunteer to provide it. Some do. 

Moreover, an ever-increasing number of scholarly journals now require disclosure 
as a condition for publication. Congress can expedite this trend by prohibiting fed- 
eral agencies from basing risk assessments on studies published by journals that do 
not practice full disclosure. Researchers who want their work to influence policy will 
seek publication in journals that require disclosure. 

Require That Any Study Used as the Basis for a Risk Assessment or Component 
Thereof Adhere to Substantive Information Quality Standards 

Information quality standards, particularly the standards of presentational and 
substantive objectivity, should not apply to all scientific research. Exploratory, hy- 
pothesis-generating research often has merit, but by its nature it often cannot com- 
ply. However, hypothesis-testing research should always comply, particularly if it is 
going to be used for risk assessment. By requiring crucial studies to adhere to sub- 
stantive information quality standards, much of the controversy over study selection 
could be eliminated. 

Notice that I would not require prior publication in a peer-reviewed journal or 
give special weight to such studies. Journals publish studies for many reasons, some 
of which are incompatible with their use in risk assessment. Full disclosure is a 
much better threshold requirement. Deference should be given to studies that, after 
full disclosure, have been reproduced and not refuted. 

Require That Agency Risk Assessments or Components Thereof Adhere to Substantive 
Information Quality Standards 

While it is crucial that key studies adhere to information quality standards, it is 
not sufficient. Considerable analysis is performed subsequent to the selection of key 
studies, so it is essential that information quality standards also apply to risk as- 
sessments and other derivative work products. 

In practice, this would mean that cancer risk assessments (including those con- 
taining unit risk values) and noncancer risk assessments (including those containing 
Reference Doses or Reference Concentrations) would have to adhere to the informa- 
tion quality paradigm. 

In the short run, this would be very difficult for ERA because, as I noted above, 
it is the published policy and practice of the Agency not to produce objective risk 
assessments. In the long run, however, this requirement would unleash a torrent 
of new research into more objective risk assessment methods. Currently, there is 
very little “market demand” for objective methods because ERA is essentially a 
monopsonist in this “market.’ That is, ERA is the only buyer; as long as ERA does 
not want objective risk assessment methods, the market will not supply any. 

Enforcement 

If Congress were to require ERA research programs to adhere to information qual- 
ity principles and standards, it would have to devise a way to enforce this require- 
ment. We know that hortatory appeals and executive certifications do not work. We 
also know that inviting judges to “do science” cannot be much of an improvement, 
for they are just as susceptible to the temptation to politicize science. Even if judi- 
cial review never erred, it also would be an expensive remedy that only a few could 
utilize. 

One way to reduce the cost of judicial review is to narrowly tailor it to take ad- 
vantage of the courts’ comparative advantage in administrative procedure. Thus, 


sial among recipients of federal research funds who considered the data they collected to he their 
private intellectual property. 
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courts might be authorized to render opinions on agency adherence to published in- 
formation quality principles and practices, but they must be kept away from sub- 
stantive scientific disputes. 

Peer Review 

Several specific reforms of EPA peer review could be considered. 

Explicitly Require Peer Reviews to Address Information Quality 

The reforms recommended above in the section on information quality would go 
a long way to solving this problem. They would make clear that adherence to infor- 
mation quality principles and standards is not optional for studies on which EPA 
intended to base a risk assessment, or for risk assessments themselves. 

As I noted above, EPA’s Peer Review Handbook gives short shrift to information 
quality. Congress could remedy this by explicitly requiring peer reviews to include 
rigorous information quality review. This should be done early in the process so that 
EPA does not commit itself to basing risk assessments on noncompliant studies. 
EPA could be sure early on that the studies on which it intends to rely are fully 
compliant and will not be the subject of a spurious later controversy. Information 
quality review also should be done later to ensure that subsequent analyses per- 
formed by the Agency also comply. Waiting until EPA has already published a draft 
risk assessment may be too late, for by that time Agency risk assessors often have 
dug in their heels. 

Considerable effort would be needed to train scientist-peer reviewers in informa- 
tion quality principles and standards, or alternatively, establish information quality 
as a distinct discipline that must be represented on every peer review panel. I prefer 
training scientist-peer reviewers so that they become better equipped to detect infor- 
mation quality errors as a regular part of their own professional discipline. This has 
external benefits insofar as it would introduce concern for information quality into 
journal peer review, and thus into scholarly research destined for journal publica- 
tion. 

Strictly Limit Scientific Peer Reviews to Science 

It might seem superfluous to make such a requirement explicit, but the record 
shows that it is needed. Peer reviewers have incentives to scientize policy, and EPA 
staff have incentives to ask peer reviewers to conduct their reviews in ways that 
at least implicitly ratify embedded policy decisions. By strictly limiting scientific 
peer reviews to science, it would be much easier to discern when any actor in the 
peer-review process — EPA staff, peer reviewers, and public commenters alike — has 
exceeded the charge. 

At a practical level, this would mean removing so-called “science policy” issues 
from peer review. This is highly desirable, for it is within the domain of “science 
policy” that politicization and scientization are most likely to occur. Also, removing 
“science policy” would make peer review a much easier task for scientists to per- 
form. It would improve the scientific quality of the peer review charge, for controver- 
sies over embedded policy choices within the charge would go away. 

If policy issues were removed from the scope of scientific peer review, the impor- 
tance of balancing bias among members of a peer review panel would appreciably 
diminish. Instead of worrying about balancing different policy views, greater atten- 
tion could be devoted to ensuring that peer review panels have diverse intellectual 
perspectives. When there is a coincidence of intellectual interests among peer re- 
viewers or between the panel and the Agency, as the current regime encourages, 
the result can be an echo chamber. 

Make the Selection of Reviewers and the Charge Independent of the Agency 

It’s a well-known secret that the ability to select peer reviewers and write the 
charge creates the opportunity to control the outcome. For this reason, EPA should 
not control the charge and peer reviewers should not be selected by EPA or its con- 
tractors. In its Peer Review Handbook, EPA displays a high degree of skepticism 


®®The echo is deafeningly loud when peer reviewers also share the same policy or “science 
policy” views as Agency staff — yet another good reason for strictly limiting scientific peer review 
to science. 

^'^It should be expected that contractors who want to maintain their business relationships 
with EPA are cognizant of EPA’s desires with respect to panel selection. 
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about external parties conducting peer reviews of their own work products. It is 
therefore hardly unreasonable for others to be similarly skeptical of peer reviews of 
EPA work products conducted by EPA. A simple expedient might be to establish 
and maintain lists of qualified, independent panel members for each discipline and 
select the requisite number of members from each list by lottery. 

In 2006, Regulatory Checkbook organized and conducted a scientific review that 
I believe follows another superior model that EPA could adopt. We followed OMB’s 
draft peer review guidelines, which were much stronger than the final version. We 
strictly limited the review to science — where possible, only primary scientific re- 
search was considered — and excluded all manner of policy considerations, such as 
the derivation of a unit risk factor. We focused on just four major scientific ques- 
tions, thus conserving resources to address only the most important issues, with a 
separate peer review panel for each. Rather than control information exchange, we 
delegated that responsibility to universally respected, bona fide subject matter ex- 
perts. So long as it did not stray into policy, we encouraged open discussion among 
all participants, including members of the public. Finally, to avoid any inter- 
ference by the sponsors, we established a Planning Committee whose role was to 
select the issues to be examined, select the subject matter experts and peer review 
panelists, write the charge, and coordinate the submission of the final reports for 
consideration by a scholarly journal subject to another round of peer review. 

Federally Chartered Advisory Committees 

A key lesson for Congress from the CASAC experience is to refrain from asking 
advisory committees to perform tasks that are inherently in conflict, such as con- 
ducting scientific review and giving policy advice. 

Where this cannot be avoided, such as existing committees whose charters it is 
impracticable to change, advisory committees should be required to abide by rel- 
evant Red Book recommendations. They should “establish and maintain a clear con- 
ceptual distinction between assessment of risks and consideration of risk manage- 
ment alternatives,” and ensure that their reports “clearly distinguish between the 
scientific basis and the policy basis” for their conclusions and recommendations. 
This can be easily enforced, such as by authorizing the EPA Administrator to ignore 
reports from advisory committees that demonstrably do not comply. The threat of 
being ignored is a powerful incentive. 

What Agency Officials Can Do Without Congressional Action 

I do not want to convey the impression that nothing can be done unless Congress 
acts. This is clearly not true. Obviously, EPA officials could, if they wanted to, insist 
that staff adhere to applicable information quality principles and standards. EPA 
officials could, if they wanted to, direct the Science Policy Council to amend the Peer 
Review Handbook to explicitly include information quality review. They could, if 


See U.S. Environmental Protection Agency (2006, p. 72). 

®®EPA appears to object even to peer reviews paid for by third parties where there is ample 
evidence of independence or no evidence of third-party control. 

Office of Management and Budget. “Proposed Bulletin on Peer Review and Information 
Quality.” Federal Register, 2003, 68, 54023-54029. 

®^We did not follow EPA’s practice of limiting the participation of independent experts to 
staged five-minute didactic presentations. 

Richard B. Belzer, James S. Bus, Ercole L. Cavalier, Steven C. Lewis, D. Warner North and 
Richard C. Pleus. “The naphthalene state of the science symposium: Objectives, organization, 
structure, and charge.” Regulatory Toxicology and Pharmacology, 2008, 51(2(1)), 1—5; Kenneth 
T. Bogen, Janet M. Benson, Garold S. Yost, John B. Morris, Alan R. Dahl, Harvey J. Clewell 
III, Kannan Krishnan and Clurtis J. Omiecinski. “Naphthalene metabolism in relation to target 
tissue anatomy, physiology, cytotoxicity and tumorigenic mechanism of action.” Regulatory Toxi- 
cology and Pharmacology, 2008, 51(2(1)), 27—36; David Brusick. “Critical assessment of the ge- 
netic toxicity of naphthalene.” Regulatory Toxicology and Pharmacology, 2008, 51(2(1)), 37-42; 
David Brusick, Mitchell S. Small, Ercole L. Cavalieri, Dhrubajyoti Chakravarti, Xinxin Ding, 
David G. Longfellow, Jun Nakamura, Eleanor C. Rogan and James A. Swenberg. “Possible 
genotoxic modes of action for naphthalene.” Regulatory Toxicology and Pharmacology, 2008, 
51(2(1)), 43-50; Fumie Y. Griego, Kenneth T. Bogen, Paul S. Price and Douglas L. Weed. “Expo- 
sure, epidemiology and human cancer incidence of naphthalene.” Regulatory Toxicology and 
Pharmacology, 2008, 51(2(1)), 22-26; D. Warner North, Kamal M. Abdo, Janet M. Benson, Alan 
R. Dahl, John B. Morris, Roger Renni and Hanspeter Witschi. “A review of whole animal bio- 
assays of the carcinogenic potential of naphthalene.” Regulatory Toxicology and Pharmacology, 
2008, 51(2(1)), 6-14; Paul S. Price and Michael A. Jayjock. “Available data on naphthalene expo- 
sures: Strengths and limitations.” Regulatory Toxicology and Pharmacology, 2008, 51(2(1)), 15— 
21 . 
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they wanted to, insist that Agency peer reviews comply with the Peer Review Hand- 
book. 

EPA officials also could, if they wanted to, require Agency peer reviews to be 
strictly limited to science. It would take hardly any effort at all for EPA officials 
to modify the charters of Agency advisory committees and specifically include within 
each a requirement to abide by the Red Book. 

In short, most of the reforms I have proposed actually require Congress to do any- 
thing. The reason why the Subcommittee is conducting oversight and considering 
legislation, however, is that EPA officials — officials appointed by Democratic and 
Republican presidents alike — have not made any of these reforms. 

Final Remarks 

My diagnosis of the problems afflicting EPA science is not novel; indeed, I have 
specifically cited papers published in 1986 that make many of the same points. 

To the best of my knowledge. Congress has never politicized EPA science by, for 
example, requiring it to estimate risk inaccurately or in a misleading way. These 
are things EPA has done on its own, often by misusing the tools of risk assessment 
(the estimation of what risk is) to justify particular risk management decisions (the 
policy determination of what risk ought to be). 

In this way, EPA risk assessors and other staff have scientized policy and politi- 
cized science. They have scientized policy by claiming that science can answer ques- 
tions that science can inform but not decide. They have politicized science by choos- 
ing not to estimate risk accurately. By scientizing policy. Agency risk assessors and 
other staff have taken away from Agency officials the authority and responsibility, 
delegated by Congress, to make policy decisions. They take away from policy offi- 
cials alternatives that are well within the range of plausible interpretations of their 
statutory directives. 

The remedies I have proposed should not be controversial if the goal is to improve 
scientific quality while preserving the Agency’s legitimate discretion under the var- 
ious laws Congress has directed it to implement. They are grounded in the ideals 
of the National Academy’s 1983 Red Book, yet recognize that the Red Book model 
has either failed or cannot be implemented. Instead of “establishing and maintain- 
ing a clear conceptual distinction between assessment of risks and the consideration 
of risk management alternatives,” I believe it is time to effect a full and complete 
separation. I believe this is essential to restore science to its rightful place, freeing 
it from politicization, while at the same time aggressively policing the boundary be- 
tween science and policy to ensure that policy making also is free from scientization. 

Some recommend removing risk assessment from EPA, believing that it is simply 
not possible for science to be performed within “the political cauldron” of EPA be- 
cause its “messy mix of politics, policy, economics, law, interests, and values” make 
it “not a good environment in which to develop and evaluate science.”®® I under- 
stand the sentiment but I am not ready to give up, nor is it clear to me that giving 
up is a realistic option. 

Thank you again for the opportunity to testify today on this important subject. 
I would be pleased to answer any question that members of the Subcommittee 
might have. 


®^In its 2004 report explaining and defending its risk assessment policies and practices, EPA 
staff say that Congress has, in fact, directed EPA to use risk assessment methods that are “pro- 
tective” (i.e., tend to overstate risk). See National Research Council (1983, pp. 151, 153). How- 
ever, the report does not provide a single example of a statutory provision requiring EPA to esti- 
mate risk in a biased manner. Every example given is either irrelevant to the question or it 
conflates risk assessment with risk management. See ibid., pp. 14—16. 

®^U.S. Environmental Protection Agency Office of the Science Advisor (2004,p. 14). “Congress 
establishes legal requirements that generally describe the level of protectiveness that EPA regu- 
lations must achieve and, infrequently. Congress imposes specific risk assessment require- 
ments.” “EPA seeks to adequately protect public and environmental health by ensuring that risk 
is not likely to be underestimated” (emphasis in original). 

®®EPA staff deny this, but unconvincingly. “[A]ny science policy position or choice used in the 
risk assessment process does not direct the risk assessment itself toward a specific risk manage- 
ment decision, e.g., the use of a specific risk estimate,” they write. “Rather, the risk assessment 
informs the decision maker about the potential risks and uncertainties around the risk esti- 
mate(s). These characterized risks are then considered in light of the other factors before a deci- 
sion is made.” (ibid., p. 13). Except that it misinforms decision makers, making it harder for 
them to take account of “other factors.” 

®®U.S. Environmental Protection Agency Office of the Science Advisor (2004,p. 11). Professor 
Marchant advocates removing the production and review of science from EPA’s jurisdiction: “it 
would be best if the science was developed and evaluated separately, and in particular in a sepa- 
rate institutional context, from the more political decision-making process.” 
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Chairman Harris. Thank you very much. 

I now recognize our fifth witness, Dr. Jerald Schnoor of the Uni- 
versity of Iowa, Department of Civil and Environmental Engineer- 
ing. 

STATEMENT OF DR. JERALD SCHNOOR, 

ALLEN S. HENRY CHAIR IN ENGINEERING, 
DEPARTMENT OF CIVIL AND ENVIRONMENTAL ENGINEERING, 
UNIVERSITY OF IOWA 

Dr. Schnoor. Good morning. Chairman Harris, Ranking Member 
Miller, distinguished Committee Members. Thank you for the 
chance to testify before the Subcommittee. I am Jerry Schnoor, Pro- 
fessor of Civil and Environmental Engineering at the University of 
Iowa. My views are my own today, and I have had the pleasure of 
serving on a number of committees concerning science at EPA, and 
I come to you with the shared interest in fostering quality science. 

But what constitutes quality science? EPA’s mission is to protect 
human health and the environment from detrimental effects of pol- 
lution and other hazards. Their science should be relative to the 
mission. It should be of high quality and high priority, and it 
should be reviewed by qualified scientists and engineers constantly. 
Such high-quality science enables excellent policy decisions to be 
made by decision makers including Congress, yourselves. In addi- 
tion, EPA science should help in identifying future and emerging 
environmental issues. I can testify that the EPA and ORD offer 
world-class science in a number of areas, including especially air 
quality modeling, monitoring, and development of emission data- 
bases. 

Improvements in air quality over the past 40 years are remark- 
able and a testament to the good science at EPA. Please see figure 
1. In the top panel here, increasing population and consumption as 
measured by gross domestic product, the vehicle miles traveled in 
America, and energy consumption are drivers. They serve to ele- 
vate the emissions in the United States. If one wants to keep up 
with the ever-increasing pollution from these drivers, it requires in- 
creasingly stringent regulations just to maintain the status quo. 
Eor the most part, U.S. greenhouse gas emissions — that is CO 2 
emissions on the slide — have mirrored the increasing U.S. popu- 
lation, which continues to grow at about one percent per year in 
figure 1. Notice all three lines in the middle are in lockstep. 

But the surprising news from figure 1 is that in the bottom 
panel, six aggregate air pollutants have been reduced by 41 percent 
over the past 18-year period. This illustrates a tremendous success 
story, which constitutes lives saved, better respiratory health for 
Americans, and billions of dollars in medical costs avoided, not to 
mention clearer, purer air. 

The United States achieved these results by virtue of steadfast 
EPA adopting new rules and enforcing the Clean Air Act and its 
amendments. The Clean Air Act is the most expensive legislation 
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to enforce in the entire U.S. Code, but it has a 30:1 benefit- to-cost 
ratio still. It has saved thousands of lives and will result in the cre- 
ation of 1.5 million new jobs over the next five years for the ambi- 
ent air quality standards alone. 

The Office of Research and Development provides a significant 
portion of scientific research at EPA. In 2011, ORD, as we have 
heard, realigned their programs from 13 to six, shown on this slide. 
The realignment was in concert with peer review provided over the 
past years by both the Science Advisory Board and the Board of 
Scientific Counselors. Motivation for this consolidation and realign- 
ment of programs reflects an emphasis on integrated 
transdisciplinary research, multipollutant exposures, and sustain- 
ability. These are not new programs but rather they represent a 
new way of thinking within ORD, and I believe considerable 
synergies may be realized in combining research into the four pro- 
grammatic areas shown on the right-hand side of the slide and the 
two smaller programs in Homeland Security and human health 
risk assessment, also on the right. 

As a member of EPA Science Advisory Board and several NRC 
committees concerned with EPA research, I can assure you that 
EPA is transparent and heavily peer reviewed already. The entire 
scientific process from major reports to published research journal 
articles, from their labs, centers, and divisions to the proposed reg- 
ulations, all are reviewed by SAB, BOSC, CASAC and other enti- 
ties. If anything, I would say the scrutiny and accountability of 
EPA has increased in recent years, based on my own experience 
serving on those committees. 

Thank you very much for the chance to testify. 

[The prepared statement of Dr. Schnoor follows:] 

Prepared Statement of Dr. Jerald Schnoor, 

Allen S. Henry Chair in Engineering, 

Department of Civil and Environmental Engineering, 

University of Iowa 

Good morning. Chairman Harris, Ranking Member Miller, Distinguished Com- 
mittee Members, ladies and gentleman. Thank you for the chance to testify before 
the Subcommittee. I am Jerald Schnoor, Professor of Civil and Environmental Engi- 
neering at the University of Iowa and Co-Director of the Center for Global and Re- 
gional Environmental Research. I am also Editor-in-Chief of the American Chemical 
Society journal, Environmental Science and Technology, a leading journal in envi- 
ronmental science and engineering. I have had the good fortune to teach and per- 
form research in the environmental area for over 35 years. During that time, I 
served as the Chair of the Board of Scientific Counselors for EPA Office of Research 
and Development (ORD) from 2000-2004, and more recently as a member of the 
Science Advisory Board (SAB) to EPA. I also am a member of the National Academy 
of Engineering and, as such, have served on a number of National Research Council 
committees of the National Academies, including one which I am chairing now on 
science for EPA’s future. So I come to you with shared interest in fostering quality 
science at EPA, and I have organized my testimony in response to the questions 
posed to me in the invitation letter from Chairman Harris dated January 25, 2012. 

What constitutes quality science to support EPA’s mission? EPA’s mission 
is to protect human health and the environment from detrimental effects of pollu- 
tion and other hazards. Thus, EPA’s science should be relevant to its mission; it 
should be of high quality and high priority; and it should be reviewed by qualified 
scientists and engineers. Such high-quality science enables excellent policy judg- 
ments to be made by decision makers. In addition, EPA science should help to iden- 
tify future and emerging environmental issues. 



117 


One recent example which illustrates how quality science can help to inform pol- 
icy even in a time of crisis involved the Macondo oil spill on April 20, 2010, and 
the subsequent release of almost 200 million gallons of oil and addition of two mil- 
lion gallons of dispersant to the Gulf of Mexico. Shortly after the accident occurred, 
EPA was asked about the toxicity of the dispersant that was chosen to break up 
the oil plume. The toxicological data on dispersants at the time were sparse, but 
EPA-ORD rapidly engaged in high-throughput testing on eight commercial 
dispersants at the National Center for Computational Toxicology in the EPA Lab 
at Research Triangle Park, North Carolina. EPA scientists learned quickly that 
Corexit 9500, the dispersant used, was comparable or relatively less toxic than other 
alternative products. In fact, EPA scientists performed the research, wrote and sub- 
mitted a scientific journal article, and subsequently published the peer-reviewed re- 
sults on June 30, 2010, only 10 weeks after the original explosion and release of 
oil (Judson et ah, 2010) — a remarkable accomplishment. Those events point to an- 
other characteristic of quality science — it should be timely. 

EPA should provide high-quality science to inform regulatory decisions. As a re- 
search engineer and editor, I can testify that the Office of Research and Develop- 
ment offers world-class science in a number of areas including air quality moni- 
toring, modeling, and development of emissions databases. Improvements in air 
quality that the U.S. has achieved over the past 40 years are a testament to the 
good science at EPA. Let’s consider air quality in recent decades in the U.S. as a 
case study for sound science to improve human health and the environment. 

Increasing population and consumption are “drivers” serving to elevate emissions 
both in the U.S. and globally. If one wants to “keep up” with ever-increasing pollu- 
tion from these drivers, it requires increasingly stringent regulations just to main- 
tain the status quo. For the most part, U.S. greenhouse gas emissions (COi-equiva- 
lents) have mirrored the increasing population which continues to grow at about one 
percent per year. Figure 1 shows the lock-step of increasing CO 2 emissions, popu- 
lation, and energy consumption in the U.S. since 1990. They track each other close- 
ly, and increasing population and energy consumption result in greater CO 2 emis- 
sions. Note that CO 2 emissions have not increased nearly as rapidly as the Gross 
Domestic Product (GDP), which indicates improved efficiency in a changing econ- 
omy. Also, the trend in the transportation sector, responsible for approximately one- 
quarter of all greenhouse gases (GHGs), shows that Americans drove many more 
miles during this period. Vehicle miles traveled increased 36% from 1990-2008, but 
the rate of release of greenhouse gas emissions due to transportation has been much 
less, especially in recent years. 

The surprising news from Figure 1 is that six aggregate air pollutants have been 
reduced by 41% over the 18-year period. This illustrates a tremendous success story 
which constitutes lives saved, better respiratory health for millions, and billions of 
dollars in medical costs avoided, not to mention cleaner/purer air. The U.S. achieved 
these results by virtue of a steadfast EPA adopting new rules and enforcing the 
Clean Air Act and its amendments. The Clean Air Act is the most expensive legisla- 
tion to enforce in the entire U.S. code, but it has a highly positive benefit-to-cost 
ratio and has resulted in lower morbidity and mortality due to lung and cardio- 
vascular disease, and the creation of many jobs by achieving and abiding by the new 
standards (CERES, 2010). CERES, an organization that articulates the views of 
major American corporations on their social responsibilities, recently estimated that 
enforcement of the National Ambient Air Quality Standards alone will result in the 
creation of 1.5 million jobs over the next five years. The country needs clean energy 
and clean air as well as high-paying jobs, and the former can augment the latter. 
In March 2011, EPA issued The Bene&s and Costs of the Clean Air Act from 1990- 
2020. According to this study, the direct benefits from the 1990 Clean Air Act 
Amendments are estimated to be almost $2 trillion for the year 2020, exceeding 
costs by a factor of more than 30:1. 
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Figure 1. Gross trends in aggregate emissions, carbon dioxide emissions, and their drivers (GDP, 
VMT, poputation, and energy consumption) since 1990 in the D.S. (Source: D.S. Environmental 
Protection Agency, Office of Research and Development, Strategic Plan, 2011) 


From Figure 2, one can see the large decline in specific emissions of NOg (30 %), 
Volatile Organic Chemicals (VOC, 53%), carhon monoxide (CO, 54%), sulfur dioxide 
(SO 2 , 55%), and particulate matter less than 10 microns (PMg, 65%) since the incep- 
tion of EPA in 1970, and the implementation and enforcement of the Clean Air Act 
and its Amendments (1967, 1976, 1990). Despite a doubling of the U.S. GDP during 
this period (and large increases in vehicle miles traveled, population, energy con- 
sumption, and CO 2 emissions), regulation of the transportation and industrial sec- 
tors has allowed a decline in emissions of air pollutants. Note, however, that the 
majority of emission reductions from 1970-2005 in Figure 2 occurred prior to 1995 
(with the exception of NO 2 ), illustrating that the rate of improvements have slowed. 
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Figure 2. Detailed trends in specific pollutants and drivers since the inception of EPA in 1970. 
(Source: TJ.S. Environmental Protection Agency, Office of Research and Development, Strategic 
Plan, 2011) 


Sometimes, there is no single entity or agency that is sufficiently interested, capa- 
ble, or funded to perform research necessary to protect human health and the envi- 
ronment from pollution. Many times, it is advantageous to form partnerships to 
combine expertise and resources. A case in point is science to understand the emis- 
sions, fate, and effects of fine particulate matter (PM 2 ) in the 1990s. EPA partnered 
with the National Institute of Environmental Health Sciences (NIEHS) and the 
Electric Power Research Institute (EPRI) to fund this seminal research. The famous 
Harvard Six Cities Study (Dockery et al., 1993) found evidence that not only lung 
cancer mortality was elevated when fine particles were prevalent in the air of U.S. 
cities, but cardiopulmonary disease also increased. However, the etiology of the dis- 
ease, the cause of cardiopulmonary mortality, was unknown. How could fine par- 
ticles cause disease, let alone death by heart attack or stroke? So it must have been 
with some trepidation that EPA began to develop regulations in 1996 to regulate 
fine particulate matter less than 2.5 microns in diameter (PM 2 ) in order to protect 
the public health. 

Today scientists have a much better idea of how fine particles can kill. In 2000, 
a reanalysis of the Harvard Six Cities Study was reported by the Health Effects In- 
stitute (HEI) and, in 2004, research was completed that validated the initial mor- 
bidity results (Krewski et al., 2004). An extended follow-up study by Francine Laden 
and colleagues was published in 2006 (Laden et al., 2006) and a summary of the 
beneficial effects on life expectancy by Pope et al. (2009). Laden was quoted in a 
Harvard School of Public Health Press Release at the time: 

“The follow-up study found that an average of three percent fewer people died 
for every reduction of one microgram per cubic meter in the average levels of 
PM 2 fine particulate matter, defined as having a diameter of 2.5 microns or 
less — narrower than the width of a human hair. This decreased death rate is 
approximate to saving 75,000 people per year in the U.S.” 


120 


That’s an example of quality science performing well — hypotheses are followed by 
hypothesis testing. Continual challenges in the peer-reviewed literature are followed 
by subsequent publication and peer review and iterated for further scrutiny of the 
results until the conclusion emerges and new questions arise. Today EPA funds re- 
search at the Rochester PM research Center and the Southern California Particle 
Center on the health effects of even finer particles, ultrafine particulate matter 
(UFP). EPA’s (2011) Progress Report states, “Ultrafine particulate matter (UFP) is 
easily transported throughout the body even beyond the cardiopulmonary system. 
Tissue and cell analysis shows evidence for the translocation of UFP to the liver, 
kidneys and central nervous system. Surprisingly, there is potential for UFP to 
cross into the circulatory and lymphoid systems, which could allow the particles to 
reach sensitive sites, such as the heart, spleen and bone marrow.” 

How does EPA currently produce quality science? Science is performed by 
the Office of Research and Development (ORD), by EPA Agency Offices (e.g., Office 
of Water), through extramural grants and contracts, and through small funding to 
the EPA Regional Offices and states (NRC, 2000). Science to inform EPA regula- 
tions is developed throughout the Agency, conveyed to the Administrator’s Office, 
and utilized accordingly. Of course, funding is provided through the budgetary proc- 
ess and Congress, and oversight is performed by GAO, 0MB, and others. EPA em- 
ploys a strategic planning process to utilize science effectively. ORD seeks to main- 
tain a balance between “problem-driven” research to address immediate policy and 
regulatory needs and “core” research in the basic environmental sciences, including 
research to understand future and emerging issues. ORD recently implemented a 
strategy to support innovation at the bench in ORD laboratories, demonstrate the 
power of trans-disciplinary research, broaden their network of problem solvers 
(crowd sourcing), and to showcase the products of such research. 

Partnerships are formed within EPA offices and across outside agencies and insti- 
tutions to perform both intramural and extramural research. Peer review of major 
products and publications is the system by which objective evaluation and criticism 
of the science occurs. Increasingly, the National Research Council of the National 
Academies has played an important role in peer review and advice to the Agency. 
Considering the importance of air quality for the Agency and the Nation, EPA con- 
tracted with NRC to produce a series of reports advising the Agency on airborne 
particulate matter in the late 1990s and early 2000s. These were viewed as quite 
helpful at a critical juncture in scientific research to inform rulemaking and policy 
(NRC, 1998; NRC, 1999). In addition, three FACA committees provide a wide range 
of important scientific peer review and advice: the Science Advisory Board (SAB), 
the Board of Scientific Counselors (BOSC), and the Clean Air Scientific Advisory 
Committee (CASAC). The SAB reviews the President’s budget request and provides 
reviews on various reports which the Agency produces. BOSC provides advice on 
management of ORD, its multi-year program plans, and reviews of its various cen- 
ters, laboratories and divisions. Of course, CASAC reviews air pollution reports, 
rules, and regulations. 

The Office of Research and Development (ORD) provides a significant portion of 
scientific research for the Agency. In 2011, EPA ORD realigned their programs from 
13 to six (Figure 3). The realignment was in concert with advice provided in recent 
years by both the Science Advisory Board (SAB) and the Board of Scientific Coun- 
selors (BOSC). Thirteen major programs proved somewhat unwieldy, and the re- 
alignment has received positive review from the SAB (SAB, 2011). Motivation for 
this consolidation and realignment of programs reflects an emphasis on integrated 
trans-disciplinary research, multi-pollutant exposures, and sustainability. These are 
not new programs but represent a new way of thinking within ORD. Considerable 
synergies may be realized in combining research into the four programmatic areas: 
Air, Climate and Energy; Safe and Sustainable Water Resources (water quality plus 
drinking water); Sustainable and Healthy Communities; and Chemical Safety for 
Sustainability; plus two smaller programs in Homeland Security Research and 
Human Health Risk Assessment (Figure 3). 

I believe ORD’s realignment is wise, moving EPA research in a new and effective 
direction. ORD is moving from a risk management paradigm, which has guided and 
influenced research over the past two decades, towards a sustainability paradigm. 
That effort will pay dividends. It is consistent with a public health approach of “pre- 
venting disease” rather than a medical approach to “treating disease” after it occurs, 
and it recognizes that environment and health are an interconnected system. And 
it follows on early pioneering research which EPA did on Pollution Prevention in 
the 1990s. Restructuring EPA’s research programs, however, is a significant chal- 
lenge to an established Agency, and ORD must effectively translate research results 
from these new amalgamated programs into scientifically informed environmental 
policy. 
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Key to ORD Program 
Acronyms 


ACE 

Air, Climate and Enersy RevSearch Program 

css 

Chemical Safety for Sustainability Research Program 

HHRA 

Human Health Risk Assessment 

HSR 

Homeland Security Research Program 

SHC 

Sustainable and Healthy Communities Research Program 

SSWR 

Safe and Sustainable Water Resources Research Program 


Figure 3. Realignment of EPA ORD’s many programs into four large programs (ACE, CSS, 
SHC, and SSWR and two small ones (HHRA and HSR), 20H 


What Improvements Are Needed for Future Science at EPA'? 

With a 41-year history, EPA finds itself in the second decade of the new Millennia 
with different challenges and variahle public support for its mission to protect 
human health and the environment. EPA has successfully controlled pollution and 
improved public health and welfare since it was formed in 1970. Success has 
stemmed largely from establishment and enforcement of its regulatory programs 
under the Safe Drinking Water Act, the Clean Air Act, FIFRA, Superfund, TSCA, 
and others. Those successes have been informed by good research, both intramurally 
and extramurally, within the Agency and outside the Agency by universities, col- 
leges, and partnering agencies/institutions. 

But EPA has been successful in reducing pollution mainly at the local scale for 
single conventional pollutants where the legislative mandate was strong. Now, our 
environmental problems are at larger scale (regional to global) and involve aspects 
without solid legislative authority (e.g., agricultural runoff, land use and climate 
change, and choice of energy systems). Some factors driving these new challenges 
to human health and the environment in the U.S. include the following: 

• Population growth and geographic shifts towards the South, West, and the 
coasts; 

• Land use change (urban sprawl, coastal development, agricultural practices); 
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• Energy choices (biofuels, shale gas by hydraulic fracturing, deep offshore oil, oil 
sands, coal bed methane, concentrated solar power, wind energy); 

• Increased consumption and technological changes (globalization of trade and 
invasive species, e-waste and complexity of new electronic devices from 11 to 
60 elements of the periodic table, new plastics and flame retardants, endocrine 
disrupting chemicals); 

• Climate change (increased precipitation intensity, changing precipitation pat- 
terns, increasing floods, droughts, forest fires, tornadoes, hurricanes). 

These factors have resulted in a new suite of emerging environmental challenges 
for EPA: 

• Air quality deterioration due to warmer, moister climate; 

• Agricultural runoff and nutrient quality criteria from climate change and land 
use choices; 

• Urban stormwater and by-pass exacerbated by sprawl and storm severity; 

• Terrestrial ecosystem degradation (loss of species such as birds, bees, butter- 
flies, bats); 

• Coastal waters ecosystems degradation (harmful algal blooms, red tides, and 
hypoxia). 

EPA’s science in the future will require a new and innovative approach to inves- 
tigating problems of broader scope where legislative mandates are not strong. Land 
use change, energy choices, coastal development, and climate change represent 
“wicked” problems of the future for which quality science is needed to chart the path 
forward. 

EPA must employ the most modern emerging technologies and tools to address 
these problems. A nimbleness and adaptability will be required to identify new envi- 
ronmental threats. Partnering and networking with other agencies, other countries, 
and U.S. citizenry to fashion creative innovative solutions to thorny problems must 
become the norm. Every form of efficiency and innovation will be necessary. Cer- 
tainly, a science budget commensurate to these pressing problems and sufficient to 
support policy decisions and regulatory actions will be needed to protect human 
health and the environment in the future. This includes better use of social, behav- 
ioral, and decision scientists who understand how to develop alternative approaches 
for desired environmental behaviors, rather than end-of-pipe command-and-control 
regulations. Sometimes there is no alternative to direct control and regulation, but 
EPA must think more creatively and seek market and behavioral solutions when 
they present themselves. 

Given the planned shift toward multi-pollutant cumulative risk assessment and 
the backlog of ten thousand chemicals that need to be assessed, there is a need to 
invest in modernizing the human risk assessment approach to move beyond the one- 
pollutant-at-a-time framework. ORD should develop a clear plan for how the outputs 
of the Chemical Safety for Sustainability (CSS) program (e.g., Tox 21, NexGen) will 
be used by the Human Health Risk Assessment program.The Safe and Sustainable 
Water Resources SSWR program will need to increase their focus on viewing water 
and wastewater holistically as an integral part of the overall water cycle. Waste- 
water is not a “waste,” but rather a resource from which we will recover water, nu- 
trients, and energy for reuse, and it will be used to make communities more socially, 
economically, and environmentally sustainable. This is in concert with EPA’s chang- 
ing role from not only a regulatory agency, but to one that promotes sustainable and 
healthy communities. Lastly, EPA should assume leadership in the social, behav- 
ioral, and decision sciences more broadly as an explicit research enterprise and 
cross-cutting strategy. Scientific research in these areas is inexpensive relative to 
the costs involved in much of the physical and biological sciences. Relatively modest 
investments in this cross-cutting domain could have large future benefits to protect 
human health and the environment (SAB, 2011). 
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Chairman Harris. Thank you very much. 

And now our final witness, Dr. S. Stanley Young, the Assistant 
Director for Bioinformatics at the National Institute of Statistical 
Sciences. 


STATEMENT OF DR. S. STANLEY YOUNG, 

ASSISTANT DIRECTOR FOR BIOINEORMATICS, 
NATIONAL INSTITUTE OF STATISTICAL SCIENCES 

Dr. Young. Mr. Chairman, Mr. Harris and others, today I am 
here to speak to making data sets used in papers supporting regu- 
lation by the APA publicly available. 

It is just good science to have data used in papers public. A claim 
can be made. Is it plausible? If the data is not available, then the 
claim is effectively “trust me” science. You might think a claim is 
made in a peer reviewed journal. Surely that makes it right. Peer 
review only says that the work meets the standards of the dis- 
cipline, and on the face of it, the claims are plausible. Scientists 
doing peer review essentially never ask for the data set. They look 
for obvious things to correct, agree or not with the claims, agree 
or not that the claims make some sense. 

How often do claims prove false or dramatically less pronounced 
in the original paper? loannidis in 2005 showed that for medical 
observational studies, claims fail about 80 percent of the time. I 
have kept an informal count of claims coming from medical obser- 
vational studies and then tested in randomized clinical trials. Over 
90 percent of the claims have failed to replicate. Yes, 90 percent 
failure rate. I refer you to a recent paper that covers these findings, 
Karr and Young, 2011. 

There are a number of technical systems and reasons for the 
high failure rate, which I will not deal with here. I will say that 
the work of Congress and the work of regulatory agencies often de- 
pend on valid science. With the best of intentions and incorrect sci- 
entific claims, you can make spectacularly bad decisions. 
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To give a historical medical example, two very large observa- 
tional studies made the claim that vitamin E will protect against 
heart attacks. Several large, randomized clinical trials did not sup- 
port those claims. Hundreds of millions of dollars were spent on the 
randomized clinical trials. 

My goal here is to suggest several things that can be done to im- 
prove the situation. Any regulation that depends on epidemiology 
studies, for example, formaldehyde, should make data sets public. 
The ACS CPC II data set that is being relied on for air pollution 
regulations should be public. 

It makes sense to fund the data generation and the data analysis 
separately. One group collects and stages the data and posts it. 
Separate groups of scientists can be funded to analyze the data. 
Other interested scientists can analyze the data. Scientists can be- 
come vested in the claims they derive from a data set. One group 
of scientists should not own a data set. 

Making efficient running of science is a good way forward. 
Science is much more efficient if the scientists have access to the 
data used to make claims. One scientist can make a claim and an- 
other can say let’s examine the data and see if the claim is sup- 
ported. Maybe there is a problem. For example, a Duke University 
study that led to clinical trials was discovered to have data-staging 
errors, just the handling of the data. 

Perhaps the scientific analysis strategy is flawed. I examined a 
data set where a claim was made that eating breakfast cereal 
would make boy babies more likely. Examination of the data set 
showed the claim was a result of a flawed statistical analysis strat- 
egy. Evidence from medical observational studies indicates that 
claims most often fail to replicate. Environmental epidemiology 
studies are just as subject to error. 

On publication of a paper where research is used to — is funded 
by the EPA, the data should be made public. When the EPA pro- 
poses a regulation based on science, it should name the papers it 
is depending on and should make the data sets used in those pa- 
pers publicly available. The agency should want to move forward 
based on good science. Congress should want the EPA regulations 
based on good science. The EPA would be more efficient if the en- 
tire scientific process is utilized. Congress would then depend not 
only on the EPA but on the normal operating of science. Claims are 
more likely to be valid and resulting policy sensible. Let normal 
science help in the vetting process. Make the data sets available. 

[The prepared statement of Dr. Young follows:] 

Prepahed Statement of Dr. S. Stanley Young, 

Assistant Director for Bioinformatics, 

National Institute of Statistical Sciences 

I am Dr. S. Stanley Young. 

I am the Assistant Director for Bioinformatics at the National Institute of Statis- 
tical Sciences, NISS. NISS is a not-for-profit, non-governmental statistics organiza- 
tion. NISS’ mission is to identify, catalyze, and foster high-impact, cross-disciplinary 
research involving the statistical sciences. I am also the CEO of Omicsoft Corpora- 
tion, a company that designs software. 

I graduated from North Carolina State University, BS, MES and a Ph.D. in statis- 
tics and genetics. 
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I’ve worked in the pharmaceutical industry on all phases of pre-clinical research, 
first at Eli Lilly and then at GlaxoSmithKline. I’ve authored or co-authored over 60 
papers and book chapters, including six “best paper” awards. I co-authored a highly 
cited book, Resampling-Based Multiple Testing, which deals with false positives, 
among other things. I have three issued patents. I conduct research in the area of 
data mining. 

I am a Fellow of the American Statistical Association and the American Associa- 
tion for the Advancement of Science. I am an adjunct professor of statistics at North 
Carolina State University, the University of Waterloo, and the University of British 
Columbia. 

Today I am here to speak to making data sets used in papers supporting regula- 
tion by the EPA publicly available. It is just good science to have data used in pa- 
pers public. A claim may be made. Is it plausible? If the data is not available, then 
the claim is effectively “trust me” science. 

You might think the claim is made in a peer reviewed journal; surely that makes 
it right. Peer review only says that the work meets the standards of the discipline 
and that on the face of it, the claims are plausible. Scientists doing peer review es- 
sentially never ask for the data set; they look for obvious things to correct and agree 
or not that the claims make some sense. 

How often do claims prove false or dramatically less pronounced than in the origi- 
nal paper? loannidis, 2005, showed that for medical observational studies, claims 
fail about 80 percent of the time. I have kept informal count of claims coming from 
medical observational studies and then tested in randomized clinical trials. Over 
90% of the claims have failed to replicate. Yes, 90% failure rate. I refer you to a 
recent paper covering these findings. Young and Karr (2011). 

There are a number of technical and systems reasons for the high failure rate, 
which I will not deal with here. I will say that the work of Congress and the work 
of regulatory agencies often depends on valid science. With the best of intentions 
and incorrect scientific claims, you can make spectacularly bad decisions. To give 
a historical medical example, two very large observational studies made the claim 
that Vitamin E will protect against heart attacks. 

Several very large randomized clinical trials did not support those claims. Hun- 
dreds of millions of dollars were spent on the RCTs. 

My goal here is to suggest several things that can be done to improve the situa- 
tion. Any regulation that depends on epidemiology studies, e.g., formaldehyde, 
should make data public. The ACS CPS II database that is being relied upon for 
air pollution regulations should be public. 

It makes sense to separately fund data generation and data analysis separately. 
One group collects and stages the data and posts it. Separate groups of scientists 
can be funded to analyze the data. Interested scientists can analyze the data. Sci- 
entists can become vested in the claims they derive from a data set. One group of 
scientists should not “own” a data set. 

Making efficient the running of science is a good way forward. Science is much 
more efficient if scientists have access to the data used to make claims. One sci- 
entist can make a claim and another can say, let’s examine the data and see if the 
claim is supported. Maybe there is a problem. For example, a Duke University study 
that lead to clinical trials was discovered to have data staging errors. Perhaps, the 
statistical analysis strategy is flawed. I examined a data set where a claim was 
made that eating breakfast cereal would make a boy baby more likely. Examination 
of the data showed the claim was the result of a flawed statistical analysis strategy. 
Evidence from medical observational studies indicates that claims most often fail to 
replicate. Environmental epidemiology studies are just as subject to error. 

On publication of a paper where research is funded by the EPA, the data should 
be made public. When the EPA proposes a regulation based on science, it should 
name the papers it is depending on, and it should make data sets used in those pa- 
pers publicly available. The agency should want to move forward based on good 
science. Congress should want the EPA regulations based on good science. The EPA 
would be more efficient if the entire scientific process is utilized. Congress would 
then depend not only on the EPA but the normal operating of science. Claims are 
more likely to be valid and the resulting policy sensible. Let normal science help 
in the vetting process. Make the data available. 

Chairman Harris. Thank you very much for your testimony, and 
now we will begin the round of questions. I will recognize myself 
for the first five minutes. And again, I want to thank you all for 
taking the time to come here and advise the Committee on such 
an important topic. 



126 


It is kind of interesting that sandwiched between the first hear- 
ing and this hearing, we had the hearing Wednesday about the 
Pavillion, Wyoming, study, which of course the ORD participated 
in, and one of the things that we discovered at the hearing was 
that in fact data was withheld until the night before the hearing, 
certainly not an example of the transparency I think some of you 
have called for. 

Let me just ask. Dr. Swackhamer, the Health Effects Institute 
has recently conducted retrospective accountability research in spe- 
cific instances to see if regulatory decisions actually produce pre- 
dicted health outcomes, and Doctor, you understand that we get 
testimony such as, you know, gee, if we just passed this rule or reg- 
ulation, you will have 200,000 less asthma cases and, you know, 
600 million less cardiac deaths and all the rest. My understanding 
is that EPA devotes a very small portion of their R&D budget to- 
ward this kind of research, and my interest is piqued because every 
single time we have had an air pollution hearing, we have been 
promised that asthma incidence would go down, and I am a physi- 
cian. You know what happened to asthma incidence in the last 30 
years, that same graph that shows that wonderful decline in air 
pollution over the last 30 years? I suggest. Dr. Schnoor, that per- 
haps you should graph the incidence of asthma. It has gone up over 
30 years. What is the EPA doing in their R&D budget to look at 
whether or not these health benefits that are claimed actually come 
to pass in the magnitude that they are claimed? Because, again, we 
have testimony that you have a 30:1 benefit ratio of doing these. 
Well, my suggestion is, great, let us spend half a billion dollars and 
we can solve our federal debt if it is a 30:1 ratio. That would be 
quite simple. As a scientist, I have to believe that is an over- 
simplification and, I suspect, an exaggeration. If we just look at 
asthma as an index case, could you tell me whether that kind of 
backward-looking evaluation would improve agency decision mak- 
ing? 

Dr. Swackhamer. What I can tell you, Mr. Chairman, is that I 
am not in charge of EPA’s budget so I don’t really know how they 
are spending their budget in terms of these kinds of studies. 

Chairman Harris. Would you recommend the Science Advisory 
Board, since the Science Advisory Board should recommend how 
science is used and they use science — see, that is the thing, and 
this is the crux of the matter. The policymakers, as has been point- 
ed out by the panel, point to science as a justification and claimed 
scientific studies that claim hundreds of thousands of less asthma 
cases, which appear not to have occurred. 

Dr. Swackhamer. Mr. Chairman, one of the things that we are 
recommending is integrating science into decision making, and 
what that means — that is kind of a fancy phrase, but what it 
means is that there is a process that both the National Academy 
has recommended, and now the Science Advisory Board in a draft 
report is recommending, and part of that process — ^you know, you 
look at a diagram — is to first formulate the problem, then do the 
science necessary to address the problem, and then complete the 
loop that you are talking about and looking at assessing whether 
the fix was appropriate and making adjustments as you go. 

Chairman Harris. Well, thank you. 
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Dr. SwACKHAMER. So we are recommending to do exactly what 
you are asking. 

Chairman Harris. Dr. Belzer, you have criticized the EPA’s ret- 
rospective look at the overall cost of the Clean Air Act, again, you 
know, this 30:1 benefit ratio. Is there a better way to assess regu- 
latory outcomes to maximize what are finite EPA resources? 

Dr. Belzer. Well, certainly, these estimates are done ex ante. 
The estimates are done before the jury comes in. It certainly would 
helpful to have retrospective analysis. I think that is a very useful 
thing. I think as a general rule, agencies don’t like to do it broadly. 
I think the NHTSA may be a good exception to that. They are a 
bunch of engineers. They really like doing that sort of thing. 

The larger problem with this set of rules is that it is EPA that 
is charged by Congress with doing the retrospective review of its 
own work and so I think 

Chairman Harris. I understand the implicit conflict of interest 
when you are charging with looking back at whether or not you 
have been effective. 

Dr. Belzer. I certainly have always been effective. 

Chairman Harris. I understand that. 

Mr. Walls, last year the EPA announced numerous changes to 
their IRIS process to respond to criticism from NAS and GAO in- 
cluding the creation of a standing IRIS advisory panel. Is the prob- 
lem fixed? 

Mr. Walls. Mr. Chairman, we think that the announcements 
from EPA are a very good step in the right direction, but I think 
we are in a situation where we really have to trust but verify. We 
have to ascertain that these changes are in fact addressing the 
problems that have been identified. I think there are still some con- 
cerns, for example, of how this new SAB committee that IRIS has 
established is going to work. I have every confidence that Dr. 
Swackhamer and her colleagues are committed to doing a great re- 
view of those assessments but I question whether or not the proc- 
ess is really independent. I think we have heard that EPA staff has 
kind of unfettered access to the reviewers. You know, that con- 
trasts sharply with peer review done by the National Academy. So 
we are encouraged. We will wait to see more. 

Chairman Harris. Thank you very much. 

I recognize the Ranking Member. 

Mr. Miller. Thank you, Mr. Chairman. 

Since there have been more than 30 years since ERDDAA was 
reauthorized, or authorized, the scientific community has had some 
time to think about it, and in preparing for reauthorizing 
ERDDAA, there have been several suggestions that would improve 
or suggestions for how to improve EPA research, to make it more 
efficient, transparent, and even more credible. One that we have 
heard repeatedly is there could be more integrated science within 
the EPA, and the National Academy of Sciences has called for a top 
science official. They say the lack of a top science official is a for- 
mula for weak scientific performance in the agency, and they sug- 
gest that Congress create a new position of Deputy Administrator 
for Science and Technology with responsibility for coordinating and 
overseeing agencywide scientific policy, peer review quality assur- 
ance. 
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Dr. Swackhamer and everyone, what is your opinion, what is 
SAB’s opinion on the advantages or disadvantages of creating a po- 
sition like that? Would that make the head of ORD — creating a 
deputy-level position make the head of ORD an obsolete position? 
Or is it realistic to think that one person, one position, would be 
able to handle the responsibility for the large test of overseeing all 
of era’s research so that we would need both a deputy and the 
head of ORD? 

Dr. Swackhamer. Mr. Chairman, Mr. Miller, the Science Advi- 
sory Board has a panel right now that is drafting a report on 
science integration at the agency, and one of the things they have 
discussed is the need for scientific leadership across the agency, not 
just ORD, which obviously has a science leader, but to improve the 
leadership across the entire agency linking the science enterprise 
both at the program offices and the regional offices with the entire 
agency to integrate across the whole agency. So we recognize the 
need for improved integrated leadership and coordination across 
the agency. We have not made a stand or made a statement or 
come to a conclusion about how to implement that in terms of 
whether it is a deputy or not. 

Mr. Greenbaum. Just to reiterate the importance of this, under- 
standing that more than two-thirds of all scientists at ERA do not 
work at ORD. They work in other parts of the program, and one 
of the big challenges is that they are all working at that place 
where they are both creating some science but they are then in- 
volved in the interpretation of that science. So making sure 
through some enhanced science leadership, there is currently a 
chief science advisor, whether that is the right mechanism or some 
other one, that there are consistent procedures for transparency, 
for peer review and various other things across the agency is an 
important factor. Whether or not the full deputy is the right ap- 
proach, as you know, there are plusses and minuses to creating a 
new senior position of that type. 

Mr. Miller. There may be more opinion than there is time, so 
could you state an opinion succinctly. Dr. Belzer? 

Dr. Belzer. I just want to comment that this idea has come up 
many times in the past. One of the things to keep in mind is that 
for a Deputy Administrator for Science to be effective, the deputy 
would have to have a very large staff and that staff would have to 
be independent of all the program offices and independent of ORD. 
So when you think about this as an idea, think about what it takes 
to fully flesh it out so that it actually has the capacity to be effec- 
tive in an agency with, what, 14,000 employees or something like 
that. 

Mr. Miller. Dr. Schnoor, did you have an opinion? 

Dr. Schnoor. Very quickly. I would agree with much of what has 
been said. I am not positive that it has got to be a deputy adminis- 
trator level, but the need for coordination of science throughout 
ERA, even down at the regional, I agree with wholeheartedly, and 
that needs to be better coordinated. 

Mr. Miller. Dr. Young? 

Dr. Young. Just a quick comment. We can talk about top down 
or we can talk about bottom up. Top down, you have a director. 
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Bottom up, if you make all the data sets available, good decisions 
will start at the bottom and the top will take care of itself 

Mr. Miller. Anyone else? Okay. I will yield back 25 seconds. 

Chairman Harris. Thank you very much, and the gentleman 
from California to my left is going to defer to the gentleman from 
California on my right while he prepares his questions. Anyway, I 
recognize Mr. McNerney from California for five minutes. 

Mr. McNerney. Thank you, Mr. Chairman. I thank my friend 
from California for letting me go first. 

I also want to thank the Chairman and the Majority Staff for 
making what appears to be a good effort at sort of a balanced ap- 
proach to this. It is a complicated subject. The EPA is a big organi- 
zation, a lot of science going on, a lot of money being spent, and 
we all want to make sure that it is done right, that the money is 
effective, we don’t make regulations that cause more problems than 
they solve, and so I really welcome the witnesses and the hearing. 

My first question goes to Dr. Belzer. In your testimony, you stat- 
ed and made a point that the EPA should be free from 
politicization. I think everybody would agree with that. There 
should be some policing of the agency to make sure there is a good 
boundary between science and policy. My question to you is, what 
about industry? Should industry have the same standards that 
draws a boundary between science and policy, or should industry 
be able to just run roughshod over policy and any decisions that 
are being made here in Washington? 

Dr. Belzer. I don’t see any reason why there would be any dif- 
ference with respect to science. The issue — I don’t think industry 
makes regulatory decisions. It is a little bit different for govern- 
ment. It has certain responsibilities that exceed those of all the in- 
terest groups that feed into it. Transparency in science and effec- 
tive peer review are probably the very best tools available for deal- 
ing with problems like scientization or politicization. I think that 
that is the best that we can do and it should be applied to every- 
one. I like the idea of having more and more competition, more and 
more people playing in this and participating and doing research, 
and let the best research win. 

Mr. McNerney. Well, I think you said the right word, the trans- 
parency. If we could make sure that industry is transparent in 
their impact on policy, then I think we would be a lot better off. 

Mr. Greenbaum, thank you for your input this morning. In your 
testimony, one of the things that was disturbing is that you men- 
tioned scientists are hesitant about getting involved in the con- 
troversial policies with the EPA or involving EPA reviewing. Could 
you give us some clue as to how we could better that situation? 

Mr. Greenbaum. Yeah, and I tried to make it very clear. I know 
that the Science Advisory Board goes out of its way to try and re- 
cruit a wide range of scientists, but if you do look at the rosters 
of people who are willing, for example, to join panels at the Na- 
tional Research Council and the group that is at EPA, is not a com- 
plete overlap. There are people who are hesitant to become in- 
volved because of the public scrutiny that comes in, the criticism, 
sometimes unfair, that comes in, and the time involved, and I think 
you need to reach out. It is particularly important to reach out be- 
yond the immediate people, for example, who are only doing work 
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in environment and health. An example of that is statisticians. 
There are a number of very good statisticians who have worked in 
the medical arena. We recruit some of those. We are ahle to do 
that, and sometimes because of who we are, we are able to get 
those people into the academies, and I think the more you can do 
that, the better off you will be because you get fresh perspectives 
and new ideas. It is not a criticism. I do think that the Science Ad- 
visory Board has tried to reach out, but there are scientists who 
will hesitate to sort of put themselves out of the frying pan and 
into the fire by going right to work for 

Mr. McNerney. Do you think the peer review process is working 
in terms of helping the EPA come to decisions? 

Mr. Greenbaum. That is a broad statement, because peer review 
applies to a lot of what the agency does, and I think it varies across 
all of the various programs and others. I think there are many 
cases certainly in the process by which the science is peer re- 
viewed, for example, in the clean air decisions. I think that the 
science part of that goes extremely well and is done as a model 
compared to, for example, the way Europe sets similar standards 
or others. But in other areas, I think there are more questions 
about exactly how that peer review operates. 

Mr. McNerney. Thank you. 

Dr. Young, you made some pretty sweeping claims about the peer 
review process. I have been involved in it, and you say that most 
peer reviewers don’t even ask for data. That is pretty — that is a 
pretty blanket statement, and I again come to the conclusion from 
your testimony that you don’t think peer review is sufficient. If 
that is the case, what do you think would be sufficient? And do you 
follow those practices in your own case? Do you follow what you 
think would be sufficient? 

Dr. Young. I always make my data sets available unless they 
are controlled by somebody else. I have made lots of data sets 
available. I am very experienced with the peer review process. I 
peer review for 10 or 15 different journals and I talk to a wide 
range of scientists that are peer reviewers. I think my statement 
is pretty accurate. The peer review of a journal article, if a scientist 
is very conscientious, he may take a day to look at a paper, he may 
take only a few hours. There is no way that that person can look 
at the data. He is trusting that the scientists on the other side did 
the work in a very legitimate way. 

Now, the other thing to say is, the 90 percent of the papers that 
I have looked at where the things did not hold up, those all came 
from peer reviewed journals. They came from JAMA. They came 
from the New England Journal of Medicine. They came from the 
best journals on the planet. Peer review does just what I said. It 
is a quick look at the data. Obvious things are fixed but the data 
is not looked at, and the evidence is, it doesn’t hold up, particularly 
in the case of observational studies, particularly in the case of epi- 
demiology studies. Ninety percent of the time, the claims do not 
hold up, and it is not just me. You guys are trusting that those 
claims are good, and I am saying with a lot of 

Mr. McNerney. So what should be done to replace peer reviews? 

Dr. Young. Peer review replacements, I am going in incremental 
steps. Make the data available. The scientific process is that then 
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other scientists can look at the data and see if they reach the same 
conclusions. Don’t rely on the peer review system, don’t rely on a 
few experts; rely on the entire scientific process. And if you make 
the data set available, that will come into play and eventually 
truth will come out. 

Mr. McNerney. May I ask another question, Mr. Chairman? 

Chairman Harris. We would like to get at least everybody in be- 
fore we break for votes and then decide. Do you have one quick 
question? 

Mr. McNerney. Well, Dr. Schnoor would like to 

Dr. Schnoor. May I just add something to what Dr. Young said? 

Chairman Harris. Sure, really quickly. 

Dr. Schnoor. As a journal editor, our policy is, and most jour- 
nals, that if the data is asked for, we do give it, but I agree with 
Dr. Young that often it is not asked for, for the reasons that he has 
expressed. 

But I would add one thing that I think is important to the story, 
and that is that usually an important paper will get eventually re- 
produced. Somebody will try to do that again, and if it fails, then 
the literature will change. I tried to represent that in my written 
testimony. 

Chairman Harris. Thank you very much. I hope they allow them 
to come in and try to verify the Pavillion findings. 

I now recognize the gentleman from California, Mr. Rohrabacher. 

Mr. Rohrabacher. Thank you very much, Mr. Chairman, and let 
me just note about the point that was just made, that I think that 
we should take that very seriously, that all data should be made 
available, not just conclusions that the people in charge of a project 
have made about the data. When we see what happened in the 
Climategate scandal, well, half of it is where we have these sci- 
entists talking to each other about how they are going to limit how 
much other people know about the data they have collected. That 
is what got everybody angry, and I know there is a group of people 
in the scientific community that want to just ignore that atrocity, 
but it is an atrocity when you hear a scientist talking about lim- 
iting knowledge, limiting the availability of knowledge to other peo- 
ple. So now on with my questions. 

I am concerned about conflict of interest in the scientific commu- 
nity, and especially in terms of the SAB and what we need here, 
let me just ask, you can have a bias towards certain businesses but 
you can have also biases towards certain positions. In the academic 
community, we find these incredible conflicts where some people 
won’t even let someone be hired by their department at a univer- 
sity if they disagree with them, and so we have to consider that 
if someone is getting, for example, ERA grants in order to prove 
something, that would suggest that they have perhaps a bias. Let 
me ask Dr. Swackhamer here on this, does SAB policy make a dis- 
tinction between the conflict of interest related to receiving federal 
grants from the EPA, for example, for research and conflicts with 
private industry? Shouldn’t both of them be considered prejudicial, 
let us say? 

Dr. Swackhamer. Mr. Chairman, Mr. Rohrabacher, yes, the SAB 
has a very rigorous process of looking at conflict of interest prior 
to every activity, so it is not just an annual kind of event. It is be- 
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fore every single activity we take on. Every member who partici- 
pates fills in a conflict of interest form and must answer questions, 
not just related to financial conflict of interest but in terms of im- 
partiality, whether they have made public statements on the topic, 
whether they have grants related to that topic. So we actually look 
at the issues, the very issues you have raised. We are actually very 
sensitive to those issues. 

Mr. Rohrabacher. So someone can receive EPA grants but — in 
other areas but they cannot serve and exercise authority in areas 
they have already received EPA grants? 

Dr. SwACKHAMER. Well, as an example, what we do is, the SAB 
does not look at individual EPA projects or individual grants. We 
look at overall programming. So if an investigator gets an EPA 
grant and also sits on the Science Advisory Board, if the grant is 
unrelated to — that specific grant is unrelated to the overall activity 
that we are reviewing, then that is not considered a conflict. But 
as an example, I have a long history of EPA funding, and we just 
recently reviewed the Great Lakes Restoration Initiative study 
plan, and I recused myself on that particular issue because the 
GLRI is such a broad program, I felt that I did have a conflict. 

Mr. Rohrabacher. Well, Mr. Walls described in his testimony 
that sometimes peer reviewers are often overly deferential to the 
EPA and the EPA staff. After all, especially if they are — if the EPA 
is giving them a grant for something, but do you think that insist- 
ing that EPA staff be present at such meetings has a chilling ef- 
fect? I will ask that to Mr. Walls. 

Mr. Walls. Yes, Mr. Rohrabacher. We do believe that there 
needs to be a higher degree of independence for the SAB in con- 
ducting these peer reviews. I think there are some practices within 
peer review in the agency and indeed across the government that 
suggest some greater links between those who are actually con- 
ducting these assessments and the peer reviewers, so we need to 
have a greater degree of independence. 

Mr. Rohrabacher. And Dr. Schnoor, is that the — okay. Good. 
You served on the SAB Board of Science Counselors, but around 
the same time you were very active in various issues. For example, 
you wrote an article about hydraulic fracturing entitled “Regulate 
Baby, Regulate,” and you also filmed a YouTube video in which you 
said that all coal-powered electric generation should be shut down. 
Do you think that someone who is such an advocate as this, doesn’t 
that represent a conflict of interest with making — giving people ad- 
vice, providing scientific advice for the EPA? 

Dr. Schnoor. Just one small correction. The issue on coal was 

no new coal-fired power plants, not to shut down 

Mr. Rohrabacher. All right. Well, that is still quite an 

Dr. Schnoor. It is a strong statement. 

Mr. Rohrabacher. It is still advocating 

Dr. Schnoor. It is a strong statement, and those are opinion 
pieces, and they are labeled as opinion pieces as such, and I do de- 
clare them in my activities on the SAB board. I do declare them 

and sometimes recuse myself from an issue directly involved 

Mr. Rohrabacher. Well, let us just note that when someone 
works for an industry, yeah, we have to understand they have a 
conflict of interest there, but if people, and especially in the aca- 
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demic world, have made a specific advocacy, that becomes part of 
their self-interest as well. 

Dr. SCHNOOR. It is, and of course, it is out there for everyone to 
see because it is freely available and published and it is disclosed 
whenever those 

Mr. Rohrabacher. As long as the same standard is for industry, 
that is fine. 

Chairman Harris. Thank you very much. 

The gentleman from New York, Mr. Tonko, is recognized for five 
minutes. 

Mr. Tonko. Thank you, Mr. Chair. 

The Subcommittee attempts to explore the areas of transparency, 
and in keeping in that vein. Dr. Belzer, your CV indicates that you 
are an independent consultant. Can you indicate for whom you con- 
sult? 

Dr. Belzer. I am not prepared to reveal the identifies of existing 
or past clients because they are covered by confidentiality agree- 
ments that I am obligated to honor. I am sorry. That is far as I 
could do on that. 

Mr. Tonko. The agreements are confidential or your relationship 
to them is confidential? 

Dr. Belzer. I am not entirely sure 

Mr. Tonko. Well, confidential materials with which you are con- 
sulting, to which you are consulting, or the groups themselves, the 
clients themselves? 

Dr. Belzer. I really don’t know. I am not sure that I am under- 
standing it clearly enough, because it seems like a fine distinction 
that is important to you but I don’t quite follow it. When I do 
things that are public or when I put my name on something, a 
piece of work that I produced, then I disclose who I did it for. 

Mr. Tonko. And the Regulatory Checkbook of which you serve as 
president is described as a nonprofit organization? 

Dr. Belzer. It sure is. 

Mr. Tonko. Who funds the Regulatory Checkbook? 

Dr. Belzer. It is funded by a small number of unfortunately too- 
small donations. We have done projects in the past and we were 
able to engineer a large project. I bring an example along. I have 
done a few things like that. But a large project will take a lot of 
donors, and those are disclosed. I brought one along. I would be 
happy to leave it for you. But the work of the project is published 
in a scholarly journal, and it has got, you know, complete disclo- 
sure of every organization that funded the work. So I have done 
other work here. I have got another thing recently published on the 
National Toxicology Program, and that is identified who funded it. 
I have done work for the government of Canada, and that is here 
as well and, you know, you are welcome to have a copy of that, too. 

Mr. Tonko. Dr. Swackhamer and Dr. Schnoor and Mr. Green- 
baum, you as a threesome seem to have the most experience on 
this panel reviewing EPA’s science, and each of you has provided 
helpful suggestions to maintain or improve science at EPA. At the 
same time, we do hear a lot of criticism of the agency’s science rou- 
tinely. And I would like to present to you — share with you a ques- 
tion and ask for a simple yes or no answer. You can elaborate after 
your yes or no. With the understanding that individual efforts may 



134 


fall short of quality goals and that things can always he improved 
upon, is the science conducted, in your opinion, by the EPA gen- 
erally of poor quality? 

Dr. ScHNOOR. I would say no, it is of good quality, but it does 
vary from topic to topic, and that means that there is always room 
for improvement. 

Dr. SwACKHAMER. I believe that the answer to that is no, it is 
not of poor quality. It is of good quality. At times it is very high 
quality. 

Mr. Tonko. Thank you. 

Mr. Greenbaum? 

Mr. Greenbaum. I would agree that the answer is no. I do think 
that there are cases where it is of — it might almost become poor 
quality, but I think also the agency has shown its ability to learn 
from mistakes, for example, in the air quality area it created an 
entire new database of the latest science on air quality and health 
which it is using and which is available to everybody who wants 
to use it to find the latest studies. That is a place where it had 
been criticized in the past and it is now doing it very well. In fact, 
it is a leader in the world in that. 

Mr. Tonko. Thank you, Mr. Chair. I will yield back. 

Chairman Harris. We are not going to have to go to vote for 
about 10 more minutes so we are going to take another five min- 
utes. We have agreed to divide five minutes per side. So let me — 
I will start and then I will turn it over to Mr. Miller. 

Dr. Schnoor, I am going to follow up a little bit about what the 
Congressman from California here on my left asked about, and 
that is the — when a scientist veers into personal advocacy, writes 
an opinion piece called, you know, “Regulate Baby, Regulate,” you 
have to look at — although it is an opinion piece, I mean, I read 
through it. I thought it was pretty interesting. Well, I thought it 
was interesting because first of all, I think you were the editor of 
the journal when you wrote the opinion piece, so that is not just 
kind of a Tom, Dick and Harry writing an opinion piece, that is the 
editor of the journal. 

Now, I am going to use a phrase, because as a scientist, you have 
to appreciate, and we brought this out at the hearing two days ago, 
that a scientist uses words very specifically. Like for instance, if 
you say “likely” that means then it probably has, you know, more 
likely than not it occurred. If you say it is the “best supporting evi- 
dence,” that means it is probably not likely but it is probably the 
best among a wide variety of alternatives. And by the way, these 
are terms that were used in the Pavillion report. You say in your 
opinion piece that for hydro fracturing causing — in addition to 
causing tap water to burn, you used the word “cause,” a very spe- 
cific scientific word that means there is a cause and effect. You 
cause it to burn. I looked through the entire literature yesterday 
and I couldn’t find scientific evidence that hydro fracturing caused 
anybody’s tap water to burn. I went back and looked at that case 
in Colorado, and I am sure you reviewed that before you wrote the 
word “cause” where in Colorado they tested that water and found 
that that water that you mentioned that comes in that Gasland 
film was in fact biogenic, not thermogenic, that in fact there is 
overwhelming evidence that it was not a result of gas coming from 
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deeper sources but it was a relative — you understand the science 
behind it. There is large series — and yet you choose to use the word 
“cause” as a scientist. Does that mean when you write an opinion 
piece, you use different words than when you write a science piece? 
Fill me in on this, because this blurring is a bone of contention on 
our oversight at the EPA. When a scientist takes work and uses 
words and then the press office uses slightly different words and 
then they kind of blend these two and then you dig a little bit, and 
oh, by the way, we don’t want to share the data until the night be- 
fore the hearing so none of our experts have a chance to look at 
it before they have a hearing. Call me skeptical. Could you address 
your use of the word “cause” in that article? 

Dr. SCHNOOR. Thank you. Chairman Harris. Yes, I can. First of 
all, the use of the word was in the context of the Gasland film as 
having caused the 

Chairman Harris. Well, if you pardon me a second, I am looking 
at the paragraph, and you said — and it is a different paragraph. It 
says — it implies that hydro fracturing caused tap water — and a 
home has exploded. Now, I won’t even get to the home exploding 
because I didn’t even know about that. 

Dr. ScHNOOR. It was in the context of referring to the Gasland 
film, and also I would like to clarify that, you know, in my job as — 
I did write that as Editor in Chief of Environmental Science and 
Technology, one of the leading journals in the world in environ- 
mental science and engineering, and I wrote it — that is a part of 
my job actually. 

Chairman Harris. Sure, and 

Dr. SCHNOOR. So in the 

Chairman Harris. I only have 1-1/2 minutes. Do you feel that 
there is scientific evidence that would permit you to use the word 
“cause”? 

Dr. ScHNOOR. To cause 

Chairman Harris. Yeah, that hydraulic fracturing 

Dr. SCHNOOR. In the context of the film 

Mr. Harris [continuing]. Caused 

Dr. SCHNOOR [continuing]. That is what I was talking about. 

Chairman Harris. So you feel that at the time you wrote in 2010 
that evidence indicated 

Dr. ScHNOOR. And regulate refers 

Mr. Harris [continuing]. Knowing 

Dr. SCHNOOR. If I could just finish answering the question, “Reg- 
ulate Baby, Regulate” refers to a lack of regulation that is laid out 
in the editorial on the oil and gas industry, which caused the 
Macondo oil well spill, 200 million gallons to the Gulf of Mexico, 
and has caused problems with pits, ponds and lagoons left over 
from natural gas 

Chairman Harris. Sure. Well, as part of our written questions, 
I am going to ask you to submit the proof of the use of the word 
“cause” because I looked over the science. I am convinced that the 
overwhelming evidence is that it didn’t cause it. 

Does anyone on the panel disagree with Dr. Young’s rec- 
ommendation that data used to justify regulations should be made 
publicly available without any — I mean, that should be taken for 
granted. Controversial data, make it publicly available, and there- 
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fore do you think the EPA should have taken well over two months 
or just about two months to release the data associated with a 
highly controversial study that was a draft report where it is the 
most important place to release preliminary data, that you are ask- 
ing people to suggest peer reviewers. Does anybody disagree that 
they should be total data transparency on an issue that important? 
Does that hand mean you disagree? 

Dr. SwACKHAMER. No, it doesn’t, Mr. Chairman. I just wanted to 
add to that. 

Chairman Harris. Sure. 

Dr. SwACKHAMER. I certainly believe that data should be made 
available for peer review and that all data that goes into making 
conclusions in a scientific study should be made available, but it is 
important that the data be through a quality review and that the 
data has been qualified assured before it is released. 

Chairman Harris. But it should have done that before you wrote 
the initial draft report, right? 

Dr. SWACKHAMER. Absolutely. 

Chairman Harris. Thank you. Okay. I am talking about the 
draft report. Thank you very much. 

Mr. Miller, we have — they just called votes but we have plenty 
of time for the five minutes or so. 

Mr. Miller. Drs. Schnoor, Swackhamer and Mr. Greenbaum, Dr. 
Young proposed funding data generation and data analysis sepa- 
rately so one group would collect and stage the data and post it 
and then a separate group of scientists with separate funding 
would analyze the data. Do you agree with that proposal? 

Mr. Greenbaum. Actually, I would not agree with that, partly 
because the design, the building of the database is a fundamental 
scientific enterprise. It is not some rote process by which somebody 
just goes out and collects data and then makes it available so that 
I do think that — and we have a data access policy, a very open data 
access policy since long before the Shelby amendment. We have al- 
ways assumed that the first data collection is being done. We are 
funding it. People are designing a study and then collecting the 
data to do it. Now, having said that, once they have had the chance 
to go through their data, make their data — analyze it, we have had 
a chance to peer review that, we very strongly believe they should 
be able to make that data publicly available and anybody should 
be able to come in and get it, and that is what I referred to in my 
testimony. We have actually put data from our studies up on the 
Web. So I don’t think it should be totally separate but I do think 
there should be access to the underlying data afterwards. 

Mr. Miller. Okay. Succinctly, Dr. Swackhamer. 

Dr. Swackhamer. I would agree very much with what you just 
heard. Thank you. 

Mr. Miller. Okay. Dr. Schnoor. 

Dr. Schnoor. Again, the policy is that if you ask for the data, 
you will get it from the journals such as ours, but I would say that 
the general impression is that the person who generated the data 
should be able to publish and work on it first, so that it is a bit 
different than what Dr. Young is proposing. 

Mr. Miller. Dr. Young, I would be delighted to, but we just don’t 
have time. 
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I now yield the balance of my time to Mr. McNerney, my time 
to Mr. McNerney. 

Mr. McNerney. Well, thank you. 

I think the peer process is kind of what we are talking about 
here then. Apparently, from what we are hearing from the panel 
this morning, most submitters do supply data with papers. Is that 
your impression? I see some shaking yes and some shaking no. 

Dr. SCHNOOR. Actually in our journal, they do, and I think Dr. 
Young agrees with that, but not every journal maybe has the same 
policy. 

Mr. McNerney. But what is coming out of this is that most re- 
viewers don’t ask for the data. So, I mean, are we in a position 
where we have to say if you want to review this, you are going to 
have to look at the data? And I don’t think that is a viable option 
because aren’t going to want to review papers if you do that. 

The other thing that kind of came out earlier was that important 
papers are reproduced, which is actually a much bigger step than 
just a peer review. If you can reproduce a paper, you have done a 
lot more than just reviewing. So requiring important papers to be 
reproduced, now, that is another level of expense and delay and so 
on. Where can we go that we don’t have to take those kind of 
steps? Yes, Dr. Young. 

Dr. Young. I refer you to a study that was done in the 1950s, 
type A personality and heart attacks. We all know it is true, don’t 
we? That study was replicated twice and both times it failed. After 
the first replication, a learned group of scientists got together and 
said the second study was wrong. Well, when it was tried to be rep- 
licated a third time and it was — it didn’t replicate, it meant the 
first study itself was wrong. So a false positive occurred in the first 
step, two more steps, and it has taken 20 years and that is a leg- 
end in our time right now. I will tell you. Congressman, you don’t 
have to lay back, you don’t have to be cool. You can be a type A 
personality and you are not going to get a heart attack. 

Mr. McNerney. There aren’t any type A personalities in here. 
Don’t worry about it. 

Dr. Young. What I am saying is that if you release the data as 
soon as possible, the scientific process can take over, and we are 
talking about not just a correct answer but how fast you get it. 
Type A personality took 20 years to overturn, and it is still a leg- 
end, so we want it to be fast and we want 

Mr. McNerney. Let Mr. Greenbaum answer. 

Mr. Greenbaum. Science is a messy business. There are thou- 
sands and thousands of papers published every year, and they can 
have a wide variety of results. They get peer reviewed. I did note 
that the quality of peer review can vary according to journals. 
There is a hierarchy, though, and as papers become — first of all, as 
they get replicated, and I even know of cases where EPA has fund- 
ed explicit attempts, independent attempts to replicate important 
results, they and Congress and industry came to us to re-analyze 
some key studies. As you get closer to key studies that are going 
to be making a difference in a risk assessment or in a decision on 
an ambient air quality standard, then you do need to get to the 
next level of understanding what the underlying data is, what the 
peer review is. I would agree with you, we cannot have detailed re- 
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analysis of every single paper out there or we would have paral- 
ysis. On the other hand, when we get to the really key studies, I 
think there is a higher standard, and that is part of why HEI was 
set up, hut we are not the only ones. There are mechanisms for 
doing that. 

Chairman Harris. Thank you very much. I thank the witnesses 
for their valuable testimony and the Members for their questions. 

The Members of the Subcommittee may have additional ques- 
tions for the witnesses, and we will ask you to respond to those in 
writing. The record will remain open for two weeks for additional 
comments from Members. 

The witnesses are excused with my thanks, and the hearing is 
adjourned. 

[Whereupon, at 11:12 a.m., the Subcommittee was adjourned.] 
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Answers to Post-Hearing Questions 

Responses hy Mr. Daniel Greenhaum, 

President and Chief Executive Officer, 

Health Effects Institute 


U.S. HOUSE OFREPRESENTATFV’ES 
COMMITTEE ONSCIENCE, SPACE, AND TECUNOEOGY 
Subcommittee on Energy & Environment 

Hearing Questions for the Record 
The Honorable Andy Harris 

Fostering Quality Science at EPA: The Need for Common Sense Reform Day II 

February 3, 2012 

Mr. Daniel Greenhaum 

1 . In discussing problems with the EPA Science Advisory Board process, Mr. Walls noted that 
"qualified scientists from industry should be given equal consideration" to serve on advisory 
panels. Do you support that recommendation? 

I do, and made that comment in my written testimony, so long as the scientist and his company do 
not have a direct or readily perceivable financial conflict of interest in the outcome of the 
particular science r£?v7evv at hand. For example, there are many reviews of chemical risk for 
which the knowledge and expertise of industiy scientists would he extremely valuable, and 
credentialed industry scientists should be allowed and encouraged to serve. However, in the case 
of a substance in which the scientist’s company has a direct financial interest in the outcome (e.g. 
a risk assessment of formaldehyde and a scientist that works for a major wood products company 
that uses formaldehyde based products in its production), the industry scientist should not be 
allowed . (Mote that this is the same standard that is applied by HEl and the National Academy 
of Sciences, and it would also apply to an academic scientist who holds any significant amount of 
stock in the same wood products company.) 

2. Many EPA science acliviiies are housed within regulatory oITices. For example, nearly half of 
EPA's laboratories do not reside in the Office of Research and Development. In your view, 
should scientific activities be organizationally insulated from regulatory activities to ensure 
objectivity and balance? 

It would be difficult to entirely isolate the scientific evaluation and synthesis work conducted 
within the program offices fiom the programmatic decision process, because that would likely 
risk NOT having the best science injected at all stages of the decision making process. However, 
the research itself can and should be done independent of the programs, and as I and others said 
at the hearing, it is critical that science evaluation and synthesis conducted anywhere in the 
Agency be subjected to the same high standards of transparency' and peer review. While this 
does occur in some case now (I noted the process by which the Ojfice oJ'Air and Radiation 
NAAQS risk and policy assessment documents are intensively and publicly peer reviewed by the 
Clean Air Scienlijic Advisory- Committee) the quality' of such peer review is not uniform across 
the agency. 


3. I understand that HEI focuses primarily on air quality research. In your view, w'ould a cofounded 
organization strucaircd similar to HEI he able to provide quality assessment and reanalysis 
information for other environmental issues such as chemical assessments or focused water 
research? 

There is no fundamental harrier to applying the principles of the HEI model to many other areas 
requiring scientific research, evaluation, and synthesis of the highest integrity' and credibility. 
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The principle needs for establishing such a model are commitments by both the regulatory agency 
(EPA) and industry to: 

• support the establishment of a truly independent Board of Directors and Scientific 
Committees that can implement the scientific activities without undue involvement of the 
sponsors; 

• provide funding for the effort over a multi-year period and in a balanced form, so as to 
take the issue of funding and credibility! “off the table: and 

• regidarly contribute useful in formation on the most pressing scientific issues and 
questions that need to he addressed. 

If these conditions can be met, the HEI model could be applied to a wide range of health and 
other environmental science issues. Having said that, the HEI model may not he necessary for 
every aspect of the science underlying regulatory^ decisions (which needs to be continually 
improved within EPA), hut may best he applied to the most controversial areas of scientific 
debate, where the impacts of potential decisions can he greatest (e.g. air pollution, groundwater 
contamination from energy activities, certain chemical risk assessments). 
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U.S. HOUSE OF REPRESENTATIVES 
COMMITTEE ON SCIENCE, SPACE, AND TECHNOEOGY 
Subcommittee on Energy & Environment 

Hearing Questions for the Record 
The Honorable Randy Neugebaucr 

Fostering Quality Science at EPA : The Need for Common Sense Reform Day II 

February 3, 2012 

Mr. Daniel Greenbaum 

1. What type of research does the EPA currently conduct to confirm predicted health outcomes 
of previously promulgated rules? In what ways can w'e improve existing efforts to examine 
the real effects of EPA rules, and w'hat else can the Agency be doing to increase 
accountability for health and environmental predictions? 

EPA has engaged in some significant efforts to re-visit whether actions they have taken have 
had the health benefits that they predicted they would have, most notably the mandated 
“Section 812” analyses in the Clean Air Act that require them to look hack at the costs and 
benefits of the Act. However, this effort, and most others, have not been based primarily on 
actual measurement of changes that have occurred, but on modeled estimates of the changes 
that ore likely to have occurred- which, although done with significant peer review, cannot 
actually measure what has happened. 

EPA has, along with HETs industry^ sponsors, strongly supported HET's efforts to conduct 
“health outcomes '' or “accountability” research which attempts to actually measure whether 
there have been improvements in air quality and health in response to EPA rides and other 
actions. To further enhance this effort, EPA could (a) build into every major new rule a 
“health outcomes research plan” which would plan out the collection of air quality and 
health data from before the rule is implemented to an appropriate period of time after 
implementation and (b) engage HE! and other organizations in producing this research in an 
independent fashion so as to ensure it is a fair and open evaluation of the rule 's 
effectiveness. 

2. Are there any legitimate concerns about making public the data sets that attempt to justify 
EPA regulations? Are there real concerns about confidentiality, and if so, do they outw’cigh 
the necessity of accountability, transparency, and open and sound scientific examination? 

There are legitimate concerns about ensuring that the personal medical and other data 
contained in the records of subjects involved in research (e.g. members of a population of 
people being studied in an epidemiologic study) are never divulged and linked to that 
individual. However, there are readily available mechanisms for masking all personal detail 
so that this breach of privacy would never occur, and this should allow academic 
inwestigators, government agencies, and private companies to make the data underlying their 
studies available. (Note that these procedures may result in slightly smaller data sets, as the 
masking of individual identity is harder ij the people come from small communities (e.g. 
where there may only he one death each day and it would be possible to link that death to a 
particular individual). But that should not undermine the value of the broader data set.) 
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U.S. HOUSE OF REPRESENTATIVES 
COMMITTEE ON SCIENCE, SPACE, AND TECHNOLOGY 
Subcommittee on Energy & Environment 

Hearing Questions for the Record 
The Honorable Brad Miller 

Fostering Quality’ Science at EPA: The Need for Common Sense Reform Da}> II 
February 3, 2012 
Mr. Daniel Greenbaum 

EPA ORD Realignment 

1 . In the realignment of the Office of Research and Development programs, has the reduction 

and integration ofthe number of programs down from 1 3 to 6, proven to be a more efficient 
and effective strategy for EPA-ORD research activities? I low has this alignment improved 
ORD's coordination with the program and regional offices? What more can and needs to be 
done to strengthen the EPA overall research enterprise? 

It is still early in the implementation of this new realignment, but it appears to be a much 
better and integrated way to plan out, on a multi-year basis, research strategies for a series 
of highly interrelated topics (e.g. Air Quality. Climaie, and Energy all in one plan). It is not 
yet clear how this M-ill change and/or improve the coordination with the program and 
regional offices, and both the EPA Science Advisory Board and a separate National 
Research Council committee on Science for EPA 's Future are examining how this can be 
better integrated agency-wide. 

This issue - of better integrating the ORD science and scientific activities across the agency 
and the assurance ofthe same level of scientific transparency and quality’ in all places - is the 
single largest challenge and opportunity for ensuring that EPA science is enhanced acroi^s 
the board. 
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Responses by Dr. Deborah L. Swackhamer, 

Professor, Environmental Health Sciences, 

University of Minnesota 

and Chairwoman, EPA Science Advisory Board 

U.S. House of Representatives 
Committee on Science, Space, and Technology 
Subcommittee on Energy and Environment 
Response to Hearing Questions for the Record 
“Fostering Quality Science at EPA: Need for Common Sense Reform Day II” 

By Deborah L. Swackhamer, Ph.D. 

Professor, University of Minnesota 
Chair, US EPA Science Advisory Board 


Hearing Questions from the Honorable Andy Harris: 

I . The !dipartisaii Policy Center's 2009 report, "Improving the Use of Science in Regulatory 
Policy" 'was mentioned during the hearing. This report made several suggestions that may be 
asefitl in guiding this Subcommittee's efforts to reform rsgiiiatoiy science, including: 

• "Studies used in the formulation of regulation should be subject to data access ' 

requirements equivaiait to those under the Data Access Act. " ‘ 

• "The process of conducting literature reviews" and "the process of naming advisory 
committees" should be made more transparent, 

• "Agencies should avoid turning repeatedly to the same scientists for sendee on advisory ' 
committees,'' 

• Executive branch agencies need to "help clarify for both ofBcials and the general public 
which aspects of disputes are truly about .scientific results and which concern policy." . 

• "Policy makers should be wary of conclusions of risk that are expressed as a single 

number," 

• "Federal agencies need to experiment with ways to increase the number of scientists 
who participate in peer review." 

• "In presenting the conclusions of literature reviews, agencies and their scientific 
advisory committees need to be as open and precise as possible in discussing levels of 
risk and uncertainty.'' 

Do you agree with any or all of these leconunendations? Do you have any additional 
comments or advice in pursuing these goals? 

The Bipartisan Policy Center’s 2009 report was directed to the entire Federal Executive branch, 
and not to EPA specifically. The EPA S AB was not asked to review or advise the report for the 
EPA Administrator, and so cannot provide comments on the report’s recommendations. The 
Subcommittee is directed to EPA administration for the agency’s view' on these 
recommendations. 


2. In discussing problems with the EPA Science Advisory Board process, Mr. Walls noted that 
'‘qualified scientists from industry- should be given equal consideration” to serve on advisory 
panels. Do you support that recommendation? 

In my years on the SAB and as Chair, our Board and Panels have had balance, including 
membership from qualified scientists from industry. In my experience, their contributions have 
been immensely valuable. 


3. Many EPA science activities axe housed within regulatory offices. For exampICj nearly half 

of EPA*s laboratories do not reside in the Office of Research and Development. In yo ur 

vieWj should scientific activities be organizationally insulated from regulatory activities to 
ensure objectivity and balance? 

The organizational structure of EPA is the responsibility of the EPA and the SAB has never been 
asked to advise on the overall organizational strueture of the Agency. Good science and relevant 
outcomes can be found in ORD as well as in the Program and Regional offices. 


Hearing Questions from the Honorable Randy Neugebauer: 

1 . What type of research does the EPA currently conduct to confirm predicted health 

outcomes of previously promulgated rules? In what ways can we improve existing 

efibils to ex^amine the real effects of EPA ruleSj and what else can the Agency be doing to 

increase accountability for health and environmentel predictions? 

The SAB is supportive of evaluating health outcomes as a result ofEPA actions. The SAB 
recommends the need to do such evaluations to "close the loop” and then make appropriate 
adjustments as needed in its recent draft report on the integration of science and decision-making 
at EPA\ However, the SAB recognizes that conducting such evaluative studies can be very 
expensive and difficult to conduct. 


2. Are there any legitimate concerns about making public data sets that attempt to justify 

EPA regulations? Are there real concerns about confidentiality, and if so, do they 
outweigh the necessity of accountability, transparency, and open and sound scientific 
examination? 

The SAB recommends that literature and data used by EPA be peer-reviewed and made available 
to the public. When the SAB conducts peer reviews and evaluations, it prefers to review all data 
associated with the document in question. It is my experience that EPA makes its best effort to 
provide all data to the SAB, subject to ethical and legal restrictions. 


^ ^ Science Integration for Decision Making at the U.S. Environmentai Protection Agency (EPA) fdraft January 5, 
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Questions from the Honorable Brad Miller: 

EPA-SAS 

1 . Testimony at the hearing alluded to concerns tliat the SAB and EPA staff is “cozy” iu 
their peer review of EPA activities. To ensure tliat tMs is not fee cases processes are 
in pl^e to prevent conflict of interest or undue influence? 

As a FACA committee, the SAB has strict processes that must be followed in its deliberations. 
All SAB meetings (including phone calls) are announced in the Federal Register and all 
discussion, presentations, deliberations are completely open to tlie public (i.e. very transparent). 
All SAD members are required to submit a conflict of interest form for EPA evaluation for each 
new advisory activity. Wc are explicitly instructed not to discuss SAB activities outside of this 
public venue (i.e. no private deliberations) and we are explicitly instructed not to interact with 
EPA staff on the activity at hand. All questions for EPA staff outside of the official meeting must 
go through the Designated Federal Officer (DFO) who has responsibility for the SAB committee 
or panel. The SAB takes FACA, transparency, and conflict of interest very seriously. 


EPAOEDReali^ment 

2. In the realignment of the 0:fficc of Research and Development programs you both 

described in your testimony, has the reduction and integration of the number of programs. 

down horn 13 to 6, proven to be a more efficient and effective strategy for EPA*ORD 

research acti'rities? How this alignment improved ORD’s coordination with the 
program and regional office? What more can and needs to be done to strengthen the 
EPA overall research enterprise? 

The SAB is supportive of the realignment of EPA ORD programs. To date, it appears that the 
realignment has led to greater interdisciplinary' interactions and greater communication. It is too 
early to tell if it has been proven an effective strategy for research outcomes. This realignment 
should result in improved coordination with the Program and Regional offices by its very design. 
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Responses hy Mr. Michael Walls, 

Vice President, Regulatory and Technical Affairs, 
American Chemistry Council 


Responses of Michael P. Walls 
American Chemistry Council 
Hearing Questions for the Record 
March 6, 2012 


The Honorable Andy Harris 


1. The Bipartisan Policy Center’s 2009 report, “Improving the Use of Science in 
Regulatory Policy” was mentioned during the hearing. This report made several 
suggests that may be useful in guiding this Subcommittee’s efforts to reform 
regulatory science, including: 

• “Studies used in the formulation of regulation should be subject to data access 
requirements equivalent to those under the Data Access Act.” 

• “The process of conducting literature reviews” and “the process of naming advisory 
committees” should be made more transparent.” 

• “Agencies should avoid turning repeatedly to the same scientists for service on 
advisory committees.” 

• Executive branch agencies need to “help clarify for both officials and the general 
public which aspects of disputes are truly about scientillc results and which concern 
policy.” 

• “Policy makers should be wary of conclusions of risk that are expressed as a single 
number.” 

• “Federal agencies need to experiment with ways to increase the number of scientists 
who participate in peer review.” 

• “In presenting the conclusions of literature reviews, agencies and their scientific 
advisory committees need to be as open and precise as possible in discussing levels 
of risk and uncertainty.” 

Do you agree with any or all of these recommendations? Do you have any additional 

comments or advice in pursuing these goals? 

Response : 

ACC agrees with the cited recommendations of the Bipartisan Policy Center’s (RPC) report. In 
particular, ACC agrees with the recommendations that focus upon improving objective and 
comprehensive scientific analysis that is developed and used by agencies and on enhancing the 
scientific advice provided to federal programs. 

In addition to the BPC recommendations, however, there is a critical need for federal agency risk 
assessment programs to adopt a consistent weight-of-evidence framework. A weight-of-the- 
evidence approach will help ensure that all relevant and reliable data, including negative results, 
can be systematically reviewed, given appropriate w'eight, and integrated in a manner that 
provides a robust understanding of the mode of action and the potential hazards and risks that 
environmentally relevant levels of exposure could pose. 

Independent scientific peer review plays a critical role of assuring agency science work products 
are objective, comprehensive and full transparent. Scientific peer review entails a detailed 
independent evaluation of the scientific work product including the critical analysis of 
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assumptions, calculations, extrapolations, alternate interpretations, methodology, criteria, and 
conclusions. Given the pivotal role that peer review plays in agency scientific determinations, 
there is a need for to agencies to evaluate their scientific peer review programs and adopt 
improvements. Such improvements should include: 

• Providing opportunity for more meaningful public comment on draft assessments to 
include a facilitated public dialogue session with stakeholders on the key issues and also 
on the draft peer review charge questions. 

• Ensuring peer review panels are composed of experts across the necessary disciplines, 
with selection based principally on scientific expertise. Affiliation and employment, and 
funding (either past or present) that are not direct conflicts of interest should nol be used 
to exclude individuals, The procedures to appoint peer reviewers should assure both 
objectivity and lack of significant bias of the individual reviewers, and a balance of 
scientific perspectives across the peer review panel. Policies and procedures for full 
disclosure of conflict of interest and bias, as described in the NAS Conflict of Interest 
Policy, should be clearly defined and implemented. 

• Structuring the peer review process to provide suflicient opportunity for robust 
discussion and consideration ofpublic comments by the independent scientific peer 
reviewers. Peer reviewers also must be afforded ample time to devote to peer reviews. 

• Responding in writing to comments as part of the assessment revision process that 
follows the public comment and peer review phases. Where the Agency elects not to 
address a peer review finding or recommendation, or a significanl public comment, EPA 
shall issue a written justification for such action. 

• Conducting a separate and independent review of the assessment after public comment 
and peer review comments are incorporated, consistent with NAS practices for scientific 
report review of NAS reports, to ensure that the agency assessment addresses comments 
received, is supported by the totality of the scientific evidence, and is consistent with 
established report review criteria. 

• Certifying that an agency assessment reflects best available science, that public 
comments and the findings and recommendations of independent peer review have been 
addressed, and that the report is impartial and objective. 

2. Last year, EPA’s Science Advisory Board accepted comments from stakeholders on 
ways to improve public participation in the Board^s meetings and reviews, and 
made some changes to their process. Bid these changes address all of the Council’s 
concerns? 

Response 

In June 201 1, the EPA Science Advisory Board (SAB) Staff Office solicited public comment on 
ways to enhance public involvement in SAB activities. The American Chemistry Council, 
among other interested stakeholders, provided written and oral suggestions to SAB. We were 
pleased to see that the Staff Office has developed additional practices to enhance public 
involvement (see 

http://voseinite.epa. aov/'sab/sabproduci.nsl/WebSARSQ/'Pub1icInvolvemeni?OnenDocumcni 

ACC, however, continues to have concerns with the SAB process. 
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The SAB Staff Office must ensure that the SAB peer reviewers fully understand their 
independent roles as peer reviewers. In general, ACC believes that members of EPA’s SAB 
office (including the Designated Federal Officials) perfomi their review function diligently and 
appropriately. At Limes, however, it appears that peer reviewers arc overly deferential to EPA 
programs, and are reluctant to be seen as criticizing EPA program staff. Also, EPA program 
staff arc often given unfettered ability to conunent througliout the peer review meetings and their 
constant presence may have a chilling effect on frank and open discussion among the peer 
reviewers. In addition, in selecting peer review panel members, the foremost consideration 
should be given to expertise. Qualified scientists from industry should be given equal 
consideration for appointment based on the subject matter, and in accordance with applicable 
conflict of interest provisions. Authoritative bodies agree, including the National Research 
Council ofthe National Academies and the Society ofToxicology. 

ACC’s concerns arc particularly applicable to EPA’s recent announcement that it has established 
a dedicated SAB peer review panel for assessments in the Integrated Risk and Information 
System (IRIS). The National Academy of Sciences (NAS) noted in its recent review of the IRIS 
formaldehyde assessment several persistent problems with peer review of draft assessments. It is 
vital that the dedicated panel operate more independently from EPA, and create processes that 
enliance public and stakeholder confidence in SAB reviews of IRIS assessments. 

ACC is concerned that a standing panel, which uses the same reviewers for multiple assessments, 
may not be sufficiently independent from the Agency. The 0MB Infonnation Quality Bulletin 
for Peer Review (see 

hilp:/7\vww-.wh!lehouse.^ov./siies/de raLilt/files/omb/meinora'nda-'Tv200.5/m05-Q3.pd0 , states that 
for highly influential scientific assessments (such as the typical IRIS assessment): 

Agencies shall avoid repeated use of the same reviewer on multiple assessments unless 
his or her participation is essential and cannot be obtained elsewhere.” For influential 
scientific assessments, “Agencies are encouraged to rotate membership on standing 
panels across the pool of qualified reviewers. 

Similarly, the EPA Peer Review Handbook (see: 

http://hvww’.epa.ao\7peerreviewypdfs/peer _review handbook _20()6.pd'f ). states that 


There is no prohibition on using the same peer reviewer more than once on llie same product 
or for multiple products of the same office. However, it is preferable to use different people 
each time to provide a broader perspective. When using a contractor to provide peer review 
services, you should recognize that contractors may have a “poo!” of reviewers that they use 
regularly. If the same peer reviewers are used repeatedly, they may lose tlieir independence 
(or the appearance ol’ independenee) from the work product(s). If a peer reviewer is asked to 
participate in multiple reviews of the same product it should be noted in the peer review 
record. 

Later in the EPA handbook, the Agency slates: 


Although persons who arc familiar with and have a substantial reputation in the field are 
often called upon repeatedly To be reviewers, it is importanl to keep a balance with new 
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people who bring fresh perspeetives to the review of a work produet. The idea here is to 
avoid the repealed use of the same reviewer on multiple assessments unless his/her 
participation is essential and cannot be obtained elsewhere. 

We recommend that EPA follow the 0MB suggestion that “the agency rotate membership 
among qualified scientists in order to obtain fresh perspectives and reinforce the reality and 
perception of independence from the agency.” 

3. Generally speaking, do you think your members would be comfortable with an EPA 
policy to make the underlying data and models used in regulatory decisions publicly 
available? 

Response 

In general, ACC and its members would support an EPA poliey to make the underlying data and 
models used in regulatory decision making publicly available. Clearly, objective evaluation of 
data is a foundation of scicncc-bascd regulatory decision-making. The public release of data, 
methods, and models used in a scientific analysis in a fonn that allows independent peer 
reviewers, and stakeholders, to independently verily the Agency’s calculationvS (and conclusions) 
and to reanalyze the data using other statistical methods and alternative models will improve 
transparency and increase public and stakeholder confidence in the decisions being supported by 
the agency’s scientific analyses. 

In many instances it is important to evaluate plausible alternative modes of action, models and 
assumptions, the choices made among those alternatives, and impacts of one choice vs. another 
on the assessment. Without the data, the impact of key assumptions and extrapolations used by 
the Agency in the analysis cannot be readily discerned. 

In ACC’s view, it is also important to distinguish between “complete data sets” and “sufficient 
data sets.” For example, one legitimate concern with making “all” health and safety data 
“public” is data compensation. Companies develop health and safety data of this nature at 
considerable expense. If complete data sets were alw-ays made public, the “public” at large and 
compclitors could make use of it in regulatory programs — both inside and outside oiThe U.S. It 
is important that those who developed the data should be compensated for the time and expense 
of developing the data. 

Data compensation is a w^eil-recognized issue and has been addressed through some EPA 
regulations (e.g. under the Federal Fungicide, Insecticide and Rodenticide Act (FIFRA) dealing 
with pesticides) and general industry practices for sharing and being compensated for data. 

A second potential practical concern of making data sets publicly available is the potential for a 
burdensome “data dump” on the Agency. A solution to this problem was developed by EPA in 
consultation with industry' for the U.S, High Production Volume (ITPV) Chemical Challenge 
Program, which began in 1998. In the HPV Program hazard data and other relevant scientific 
infomiation for U.S. HPV chemicals (either existing or newdy acquired information) w'crc 
submitted to EPA. 
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From a practical standpoint, EPA determined that it was not reasonable to attempt to create an 
electronic version of full study reports (especially old reports from the 1960s or earlier) for the 
HPV Challenge program. See: littp:/7wv^w\cpa.gov/lipv/pubs/gencra[/robsmngd.htm . instead, 
detailed electronic summaries of full study reports w'erc prepared that contained the appropriate 
technical tnlbrmalion for that particular endpoint. The term "robust summary" was used to 
describe this technical content. 

Robust study summaries are intended to provide sufficient data/information to allow a 
technically qualified person to make an independent assessment of a given study report without 
having to go back to the full study report. They w'ere also intended to allow EPA to reach 
decisions w-hether the existing data, together with other relevant information, were sufficient or 
w^hethcr additional testing would be needed. The details of these robust summaries — “sufficient 
data sets” — also enabled the public, including environmental NGOs and animal welfare 
stakeholders, to evaluate proposed test plans in the IIPV Challenge program. 

A robust study summary therefore might be a path forwurd to ensure “sufficient data sets” are 
made public, since a robust summary reflects the objectives, methods, results, and conclusions of 
the full study report. So, w-hile there may be practical concerns about overly burdening the 
Agency with a “data dump” of “complete” data sets, (w'hich the Agency doesn’t always need to 
make its decisions), the robust study summary approach is a viable solution to this potential 
concem. 

A third potential concem relates to requirements to submit data and information before scientists 
have had a chance to analyze it and publish the data themselves. One of the witnesses at the Feb. 
3 hearing called for the scientists wEo developed data for the government to turn the data sets 
over immediately to other scientists who would analyze the data. This proposal was 
recommended as a means of avoiding problems of bias, inadequate data analysis and similar 
concerns by the scientists who collected the data. 

A requirement to submit all data immediately would remove an important incentive for scienlisls 
to conduct research and collect data. The scientific community relies on scientists to develop 
new hypotheses, design new- methods and innovative approaches to study these hypotheses, and 
to collect, analyze and publish their research results in the scientific literature. These researchers 
should be afforded tbe opportunity to anal^-zc and publish the results of tbeir research efforts 
before others are given the opportunity to analyze the data In sum, liming of the release of 
scientific data is certainly one concern with making “all” data sets which EPA might rely upon, 
publicly available. 

The Bipartisan Policy Center report mentioned in your first question (“Improving the Use of 
Science in Regulatory Policy,” available at 

httD:/7www.bipaiti$anDQlicv.Qrg/sitcs/dcta ii lt''filcs/BPC%20Scieiic c% 20Rcport%20f n l.pdf 

recommends that “studies used in the formulation of regulation should be subject to data access 
requirements equivalent to those under the Data Access Act (Shelby Amendment) and its 
implementing Circular regardless of who conducted or sponsored the study. In addition, the Data 
Quality Act and Agency implementing guidelines provide benchmarks and criteria that are to be 
followed by agencies in disseminating data and other relevant information that support 
regulatory evaluations. 
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4. A recent joint report from the EPA’s Science Advisory Board and Board of 
Scientific Counselors recommended that the Agency ‘^^include sustainability in its 
research vision” in order to allow “EPA to adopt sustainability as a core principle to 
inform decisions and actions.” Is this emphasis on sustainability appropriate for 
EPA’s research and science activities? Do you have any concerns abont this new 
mission? 

Sustainability is primarily concerned with maximizing benefit, while addressing risks of concern, 
rather than being an exercise focused mainly on achieving risk based standards. ACC believes 
that adopting sustainability as a core principle to inform decisions and actions will in fact enable 
EPA to formally incorporate consideration ofbenelits in the their risk-benellt evaluations — but 
only if it is implemented appropriately. A sustainability framework will encourage EPA to take 
on a more integrated and systems-based approach to environmental management that can be 
superior to the current patchwork approach. A recent report by the National Academy of 
Sciences (“Sustainability and the U.S. EPA,” (201 1)) outlined one possible approach to 
implementing sustainability in EPA activities. 

The ACC believes that the private sector, and our industry, in particular, has a ricb experience in 
incorporating sustainability into our business processes. We believe tlie Agency w-'ould benefit 
significantly from engaging stakeholders in developing a strategy to operationalize sustainability 
in its vision. 

5. Many EPA science activities are housed within regulatory offices. For example, 
nearly half of EPA’s laboratories do not reside in the Office of Research and 
Development. In your view, should scientific activities be organizationally insulated 
from regulatory' activities to ensure objectivity and balance? 

Response 

As one of the Nation’s most regulated industries, ACC member companies expect high quality 
science and reliable assessmail processes - to underpin elTecti ve and efficient regulatory 
decisions by the Federal government. It is critically important that EPA laboratories conduct 
research that is mission-critical and can be held to the highest standards of excellence. 

Part of ensuring this quality requires that EPA make certain that research is conducted in an 
objective manner that is not innuenced by a desire for a predetermined outcome. If we look at 
the IRTS program, there is a lack ofa consisLenl, coherent, science-based framework that binds 
the agency to an appropriate and transparent approach for weighing evidence, considering 
uncertainty, and keeping up with advances in the field. 'I hese deficits greatly impact stakeholder 
confidence in IRIS database. 

Objectivity and high quality are not necessarily a function of where an assessment program is 
located or organized. The IRIS program is housed in an Office of Research and Development 
Laboratory (the National Center for Environmental Assessment) and yet we see that the 
independence from regulatory activities has not necessarily guaranteed the objectivity and high 
quality science that all stakeholders would like to see. 
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A draft report released by the HPA Science Advisory Board (SAB) in January 2012, noted that 
scientific integration could be strengthened and that the agency’s focus on program and 
disciplinary “silos” remains a significant barrier to science integration. ACC supports measures 
to improve scientiHc integration that result in further enhancements in science based regulatory 
decisions. 
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The Honorable Randy Neugebauer 

1. What type of research does the EPA currently conduct to confirm predicted health 
outcomes to previously promulgated rules? In what ways can we improve existing 
efforts to examine the real effects of EPA rules, and what else can the Agency be 
doing to increase accountability for health and environmental predictions? 

Response 

ACC is not aware of any standardized approaches or practices used by EPA to confirm predicted 
health outcomes to previously promulgated rules. While EPA does have aceess to aneedotal 
information (for example, we know that blood lead levels in Americans decreased substantially 
after lead was removed from most gasoline), a more systematic approach to collecting and 
evaluating this infonnation would be welcomed. 

2 . Arc there any legitimate concerns about making public the data sets that attempt to 
justify EPA regulations? Are there real concerns about confidentiality, and if so, do 
they outweigh the necessity of accountability, transparency, and open and sound 
scientific examination? 

Response 

ACC’s response to Dr. Harris’ third question outlines a number of general concerns that may 
arise from requirements to make public data sets that justify EPA regulation. 

One potential disconnect in considering the release of data sets that justify federal regulation is 
that many federally supported researchers may have no idea that tlic work they arc doing may 
someday be the basis, or part of the basis, of a regulatory action by some agency before which 
they have never appeared and about which they know little. It would be natural to assume that 
there would be some resistance in the research community at a broad requirement for disclosure 
of data sets generated by researchers without prior knowledge of the possible use as a basis for 
regulation, and without an opportunity to fu 11 leverage the scientific and research potential of that 
data. 

In ACC’s view, it may be diflieult to implement a blanket requirement to disclose the data sets 
underlying regulatory programs. One approach would be to provide incentives for enhanced data 
sharing. That might include resources (or investigators to clean up data sets, meet data quality 
criteria, and submit them to a managed database. 

Finally, there are indeed important and legitimate concerns regarding the protection of 
confidentiality with respect to some data sets generated by private research. For example, much 
of the innovation in chemistry depends on confidential chemical identities, which often cannot be 
patented. ACC strongly supports transparency and the public availability of health and safety 
information about potential effects of chemicals, while still protecting competitive business 
interests. We believe these often competing interests can be reconciled by distinguishing 
between the full disclosure ofhealth and safety effects infonnation and the nondisclosure of 
legitimate confidential business information and conildential chemical identities. 
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QUESTIONS FRO M C HAIRMAN ANDY HARRIS 

1. The Bipartisan Policy Center's 2009 report, "Improving the Use of 
Science in Reguiatory Policy" was mentioned during the hearing. 
This report made severai suggestions that may be usefui in 
guiding this Subcommittee's efforts to reform regulatory science. 
Do you agree with any or ail of these recommendations? Do you 
have any additional comments or advice in pursuing these goals? 

As a preface to my reply, I wish to make three observations that span 
the range of the BPC committee report's conclusions and recommendations. 
First, there should be no question that the committee approached its 
assignment with seriousness, public spiritedness, and the best of intentions. 
It is therefore inappropriate to judge its insights and commendations based 
on any factor other than merit. 

Second, the BPC committee's task was enriched by the participation of 
individuals who had served, at one time or another, on both the analytic and 
decision making sides. This experience makes a fine illustration of Graham 
Allison's Model 3 of bureaucratic politics; "Where you stand depends on 
where you sit."^ 

Third, the good faith and rich experience of the members of the BPC 
committee do not automatically translate into practical or effective solutions. 


’■ Allison (1971). 
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Some of the committee's recommendations are basically fond wishes for 
someone — anyone — to implement by magic wand. Other recommendations 
appear to be plausible ideas that unfortunately are grounded on dubious 
premises. 

The BPC committee's recommendations should be subjected to a 
sequential three-part analysis: 

1. If the recommendation could be implemented immediately at no 
cost, would it solve the identified problem? If it wouldn't, then it is 
unclear why Congress should devote much time to it. 

2. If the recommendation could solve the identified problem, is there 
a practical strategy proposed by which to implement it? If the 
recommendation cannot be implemented, then its interest will be 
limited to academics and theoreticians. 

3. If there is a practical strategy by which the recommendation could 
be implemented, what unintended consequences could occur; 
which of them are likely; and how could they be prevented? 

With that preface, my responses to each of the bulleted inquires 
follows below. 

• "Studies used in the formulation of regulation should be 
subject to data access requirements equivalent to those 
under the Data Access Act, " 

I am generally in agreement with the objective of the BPC committee 
recommendation, but it appears to be too timid. Data access rules 
"equivalent to those" under the Data Access Act cannot solve the problem 
the BPC committee identified. Agencies can and do behave strategically to 
evade the Shelby Amendment. They often rely substantially on the work of 
federally-funded researchers but intentionally do not obtain their data. 
Meanwhile, 0MB Circular A-130 is burdensome and ineffective. In short, this 
recommendation fails the first element of my three-part test. 

A SUPERIOR ALTERNATIVE 

In my testimony before the Committee, I said the full disclosure of 
data, models, and assumptions should be required for scientific information 
that EPA (or any other agency) either disseminates in a manner connoting 
agreement or which it relies on, in whole or in part, for regulatory decision 
making. As I testified: 

Congress could relieve Federal agencies of this conundrum by 

requiring them to obtain research data if they want to use a Federally 

funded study as the basis for risk assessment. Requiring disclosure 
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imposes only trivial costs on the agencies and does not violate the 
contractual terms of any Federally-funded researcher. Mo burden 
would be imposed on anyone if the agency did not want to use a 
Federally-funded study as the basis for risk assessment, and no 
researcher would be compelled to accept Federal research funds to 
conduct a study likely to be useful in risk assessment.^ 

In my testimony I also agreed with the BPC committee that no 
distinction should be made based on the source of research funding; 

If an agency wants to rely on a study that was funded by another 
party, whether that be a state, business, trade association, or 
nongovernmental organization, nothing currently prevents the agency 
from asking that this information be supplied, nor is there any general 
legal barrier to the other party providing it. States, businesses, trade 
associations, and nongovernmental organizations that want their 
research to be used for public policy should happily volunteer to 
provide it.^ 

Even if It were true that industry-funded studies always pointed to 
lower risk and government/nonprofit studies always pointed to higher risk, 
that would not justify applying different disclosure standards. Rather, it 
reinforces the need for the same standards to apply to all scientific 
information, regardless of the source of funding or the direction in which the 
research might alter risk assessment. 

• "The process of conducting literature reviews" and "the 
process of naming advisory committees" should be made 
more transparent. 

These excerpts comes from different recommendations addressing 
different issues, though with an overlapping remedy.'* 

LITERATURE REVIEW 

The BPC committee correctly recognized that if the scientific record is 
not assembled well at the outset, the rest of the process will go badly, and 
transparency has been lacking with regard to this process. Further, the 
committee also noted that EPA's criteria for evaluating the literature lacked 
consistent principles, which is a discreet way of saying that Agency 
procedures are ad hoc, and in many cases post hoc. 


^Belzer(2012), p. 23. 

^ Ibid. 

■* Bipartisan Policy Center (2009), p. 41 [Recommendation Three] and p. 18 
[Recommendation Two]. 
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What the committee did not say, but almost surely knew, is that EPA's 
opacity with respect to conducting literature reviews is intentional. Opacity 
maximizes the Agency's policy discretion to interpret scientific information as 
it sees fit. Thus, the committee's recommendation is very nearly a request 
that EPA bind itself to interpret and use science in predictable ways. Unless it 
is required by law, this is something neither EPA nor its advisory committees 
are likely to do. Thus, it fails the second element of my three-part test. 

SELECTION OF MEMBERS OF ADVISORY COMMITTEES 

The BPC committee's recommendation for transparency in the 
selection of scientific advisors is similarly at odds with the incentives of EPA 
officials, staff, and its scientific advisors. Each has incentives that are 
incompatible with each other, and with good government. Officials and staff 
alike tend to prefer scientific advisors who agree with them and be loyal 
defenders of their parochial interests. Officials and staff sometimes agree 
with each other, but sometimes do not. Thus, conflict is built into the process 
when an agency allows third parties inside. 

It is useful to remember that advisory committees were the last 
generation's good-government solution to the problem of agencies failing to 
rely on the best available science. The BPC committee’s conclusion that the 
advisory committee process needed reform indicates that this reform has not 
been successful and it has had significant unintended consequences. Among 
those unintended consequences is a new source of pressure to scientize 
policy. 

Being a scientific advisor to EPA confers prestige and power, and for 
academics it also provides a potential trail to money in the form of research 
grants. Prestige is obvious; power arises because of the ability to influence 
policy; and research grants are the mother's milk of academia. It would be 
naive to think that scientific advisors are motivated solely by altruism. 

The BPC committee called for the process of naming advisory 
committee members being made more transparent, but it is not clear which 
problem the committee was trying to solve. The committee said there was a 
"proper" way advisory committees should be used; implied that agencies' 
actual use of them was not "proper"; and concluded that transparency in the 
selection of members would restore "propriety."^ The committee did not 
explain how this would happen. 



Ibid. p. 68. 
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Transparency is certainly a good thing, but it isn't likely to be a 
solution to the underlying problem (i.e., failure to use the best available 
science) or advisory committees' unintended consequences (e.g., 
scientization). These unintended consequences can be reduced, but not 
eliminated, by strictly limiting the role of scientific advisory committee to 
science. For both scientific and policy advisory committees, there would be 
additional benefit in making committee selections randomly from lists of 
qualified individuals, thus reducing the potential for lobbying, logrolling, or 
various forms of corruption. Ironically, this would make the selection process 
less transparent, not more so. 

RELATED ISSUES 

I wish to comment upon one of the committee's recommendations in 
this area that I find troubling: 

In general, papers in high impact, peer reviewed journals 
should be given great weight, and papers that have not been 
peer reviewed should be treated with skepticism. However, the 
quality of peer review varies widely, and Journal rankings and impact 
factors do not guarantee that peer review of a specific paper was 
performed adequately. Agencies and scientific advisory 
committees need to extend their inquiry beyond simply 
ascertaining whether a paper was peer reviewed; peer review 
is a necessary but not sufficient determinant of quality. That 
further inquiry might explore how the peer review was conducted, how 
the paper fits into the larger body of literature under review, and 
perhaps most important, the methodology behind the conclusions 
described in the paper (for example, how a cohort to study was chosen 
in an epidemiological study).® 

This advice is internally inconsistent. The committee says that studies 
published in "high impact, peer reviewed journals" deserve "great weight," 
but then cautions that the "quality of peer review varies widely," which of 
course also is true for "high impact, peer reviewed journals." If that is so, 
then what could possibly be the justification for giving deference to these 
studies? It is a short step from giving deference to studies published in 
prestigious journals to giving deference to studies authored by prestigious 
researchers. Prestige is not a predictor of accuracy, and what we ought to be 
seeking to encourage is a scientific culture in which accuracy is what leads to 
prestige. 
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The additional criteria suggested by the committee are early steps of 
the pre-dissemination reviewf required by 0MB (and EPA) information quality 
guidelines. However, the BPC committee curiously excluded federal 
information quality standards from the scope of its work — an obvious lost 
opportunity.^ 

In my testimony, I recommended that Congress require that EPA risk 
assessments, components, and studies used as the basis for a risk 
assessment or component, adhere to information quality standards.® This is 
a much better path forward. It would establish a well-defined and consistent 
performance standard for all scientific information used in support of 
regulation. It focuses on the objectivity of scientific research and its utility for 
decision making, not weak, poorly correlated proxies such as the perceived 
prestige of the journal (or the researcher). 

For this reason it is curious that OMB's peer review guidelines do not 
require adherence to federal information quality standards, even though 
information quality was advertised as their raison d'etre. Information quality 
review also is missing from EPA's Peer Review Handbook.® Apparently EPA 
does not want its scientific peer reviews to get distracted by the burden of 
ensuring that information quality principles, including objectivity, are met. 

• "Agencies should avoid turning repeatedly to the same 
scientists for service on advisory committees. " 

This advice from the BPC committee is part of a series of elements in 
its second recommendation, the purpose of which is as unclear as the theme 
linking the elements is elusive.^® Though the committee apparently found it 
easy to recommend against going to the same well too often, it did not make 
clear what might be wrong with its water. 


^ Ibid. p. 43; footnote 6. This is peculiar. No less than nine references in the 
Report's bibliography concern federal information quality standards. Even more 
curiously, this footnote does not appear to be relevant to the text to which it is 
assigned. 

® Belzer (2012), pp. 24-25. 

® U.S. Environmental Protection Agency (2006), p. 17. Nothing in the Peer 
Review Handbook explains how to actually perform an information quality review. 

Bipartisan Policy Center (2009). The BPC committee's second 
recommendation is that the Administration promulgate guidelines implementing the 
committee's list of recommendations about when to consult advisory committees, 
how to appoint them, and how they should operate. 
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There are several problems that can arise if EPA relies repeatedly on 
the same scientists for advice. The agency may prefer to retain scientists 
who share the same perspective on the agency's mission and policy direction, 
or who are more easily managed by the career staff. These scientists would 
reflect too narrow a perspective, and easily could become so powerful that 
they are (or perceive themselves to be) de facto regulatory decision makers. 
As I noted in my testimony, in 2008 the Clean Air Scientific Advisory 
Committee (CASAC) seems to have succumbed to this misunderstanding of 
its role with respect to Its review of the ozone NAAQS.^^ More recently, in its 
now-abandoned proposal to "reconsider" the 2008 ozone NAAQS, EPA 
Administrator Jackson appears to have welcomed the opportunity to implicitly 
delegate to CASAC the authority to set the standard. The scientization of 
policy is inappropriate whether it is committed by scientists on an advisory 
committee or the Administrator herself. 

At the same time, there may be good reasons for asking the same 
scientists to serve over an extended period. For example, a regulatory 
development process (e.g., the NAAQS) that takes five years may need 
multiple reviews. If the reviewers change midstream, there is a significant 
chance that the second group of reviewers will give advice that is contrary to 
that of the first. While the first group's advice might have been wrong, it's 
just as plausible that it was right and it is the second group's advice that 
isn't. The quality of EPA's science is not necessarily enhanced when it 
receives conflicting advice from multiple committees. 

In my view, when EPA gets conflicting advice, it is likely that the 
reason is not because of the length of service of certain peer reviewers and 
advisory committee members. Rather, conflicting advice arises because the 
nature of their role is conflicted. This happens when scientist/reviewers are 
asked to conduct both a scientific review (which should be neutral and 
objective) and opine on policy (which cannot be). Whether there is churn 
among per reviewers and advisory group members may not matter a great 
deal if they limit their work to science. But it could matter a lot if they are 
providing policy advice, something the BPC committee explicitly advised 
against. Thus, the more important first step is to strictly limit scientific 
reviews to science and get advisory committees out of the business of doing 
both scientific review and giving policy advice. 


“ Belzer (2012), pp, 19-20. 

U.S. Environmental Protection Agency (2009), U.S. Environmental 
Protection Agency (2010a), p. 2943. 
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* Executive branch agencies need to "heip clarify for both 

officials and the general public which aspects of disputes are 
truly about scientific results and which concern policy. " 

The BPC committee's suggestions in this section of their Report are all 
interesting and potentially very useful, but they beg the question: why 
haven't any of them already been implemented? After all, like other 
regulatory agencies EPA has been subject to centralized review by OMB's 
Office of Information and Regulatory Affairs for consistency with various 
regulatory principles for over 30 years. 

One answer is that abstract presidential directives and guidelines are 
not always consistent with the president's agenda or enjoy bipartisan 
congressional support. Even when these barriers do not exist, they are far 
from self-implementing. The OIRA professional staff is too small and its 
authority too limited to be as effective as its advocates hope and its 
detractors fear. 

Even when the stars are properly aligned, OIRA's review process is not 
structured in a manner that enhances effectiveness. One obvious example: 
OIRA review occurs too late in the process to ensure that presidentiai 
guidelines have been met. Thus, even if guidelines that the BPC committee 
considered ideal could be drafted, neither OIRA nor anyone else could 
enforce them. What the BPC committee did not say, but which has to be 
true, is that the reforms it wants the President to implement are contrary to 
the bureaucratic interests of the agencies he superintends. Note also that the 
BPC committee did not propose that OIRA's authorities be expanded enough 
to enforce them, or that a new organization be established and given this 
authority. 

There are many strategies that might be considered for ensuring that 
EPA (and other agencies) "clarify for both officials and the general public 
which aspects of disputes are truly about scientific results and which concern 
policy." In my testimony, I recommended that Congress require agencies to 
comply with Federal information quality guidelines and explicitly give the 
courts the limited authority to adjudicate adherence to these procedures and 
standards. This recommendation is much more practical and easier to 
implement than yet another unenforceable presidential guidance document. 
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It satisfies at least the first two elements of the three-part test I presented at 
the outset.^"' 

• "Policy makers should be wary of conclusions of risk that 
are expressed as a single number. " 

The BPC committee's advice here is welcome but, if anything, seriously 
understated and naive. The notion that risk can be reduced to a single 
number is a longstanding and durable myth, but unfortunately it is one that 
Congress encourages. Risk assessment is scientifically uncertain and risk is 
inherently variable across any population, but policy makers across both the 
Executive and Legislative branches persist in seeking single number (and 
single word) characterizations of risk. The use of single numbers (and single 
words) to represent or describe risk is a common way that policy issues are 
scientized — i.e., where science is used to make it appear as if no genuine 
policy issue exists. 

The proper way to report risk estimates is by first objectively 
characterizing the entire distribution, to the extent that is feasible, and then 
by conducting a rigorous analysis of the most important scientific 
uncertainties. Not only is distributional variability rarely reported and 
uncertainty analysis rarely performed, human health risk is not estimated 
objectively. If adherence to Federal information quality guidelines and 
standards were statutorily required in an enforceable manner, these 
longstanding problems would have a short shelf life. 

• "Federal agencies need to experiment with ways to increase 
the number of scientists who participate in peer review." 

The BPC committee's concerns about the effectiveness of 
governmental peer review are certainly well-founded, but I am not 
persuaded by the committee's diagnosis, which depends on several dubious 
premises and factual claims that are not well supported by empirical 


The most likely unintended consequence would be judicial interference with 
science. My recommendation does not include giving the courts the authority to 
review and substantiveiy opine on science, but the courts might not heed such a 
restriction. 

A non-EPA exampie of some interest: Congress directed the Department of 
Health and Human Services (through the National Toxicology Program) to determine 
whether substances are "known" or "reasonably anticipated to be" human 
carcinogens — special forms of the "single number" problem. However, neither of 
these conditions is discernable scientifically, and as a result the NTP's biennial Report 
on Carcinogens has little scientific merit. See Belzer (2012), pp. 19-20. 
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evidence. For example, it is not clear that peer review has ever been, as the 
BPC committee claims, "the primary guarantor of integrity in the scientific 
system."'® 

Scientific integrity has never been guaranteed by anything, and peer 
review would be a poor insurance policy. This is not because "[s]cientists 
may feel too burdened to review their colleagues' papers or may do so with 
insufficient care," or because "[p]eer review is no longer assumed to be a 
professional obligation," though both concerns may be valid. The reason is 
more mundane: the historic purpose of peer review has been to allocate 
limited research funds across competing proposals and to decide which 
manuscripts among competing submissions deserve to be published in 
scholarly journals whose pages are limited. Scholarly integrity is a product of 
training reinforced by character; it is not part of peer review. A more 
plausible reason for the decline in the quality of peer review, if indeed that 
has happened. Is that academic institutions no longer spend as much time 
inculcating integrity among junior scholars and valuing character. 

There is an increasing tendency for academic scholarship to be infused 
with policy advocacy. Whereas a generation ago, men and women chose 
scholarly pursuits to advance knowledge, it seems that an increasing 
proportion of them do so nowadays to advance hobby horse public policy 
objectives. This is a trend that so many academic institutions and 
professional societies foster that it is getting harder every day to find 
scientists to conduct peer review who are as interested in the science as they 
are in whether the science advances the achievement of their public policy 
preferences. For this reason alone, the BPC committee's suggestion that 
universities and professional societies do a better job fostering peer review 
seems unlikely to be effective. It fails at least the second element of my 
three-part test. 

A larger problem is that governmental peer review is structured very 
differently from scholarly peer review and has a completely different 
objective. Whereas scholarly peer reviewers are never selected by the 
authors of the manuscripts they review, governmental peer reviewers often 
are. Whereas scholarly peer reviewers have substantial influence over 
whether manuscripts are published, governmental peer reviewers never do. 
Whereas scholarly peer reviewers are supposed to determine whether a 


IS 


Bipartisan Policy Center (2009), p. 45 (and pullout text on p. 46). 
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manuscript deserves to be published, governmental peer reviewers are asked 
to determine whether a risk assessment is correct. 

As a result of testimony given by several of the witnesses at the 
February 3 hearing, the Committee is now aware, perhaps for the first time, 
that peer reviewers in both scholarly and governmental settings virtually 
never review a study's underlying data. The Committee also appears to have 
become aware for the first time that peer review is a poor tool for 
ascertaining whether the conclusions of research are scientifically correct. 

This misunderstanding preceded, but seems to have been exacerbated 
by, OMB’s government-wide peer review guidelines.^® The stated purpose of 
these guidelines was to provide a mechanism for pre-dissemination 
information quality review. Inexplicably, however, OMB's guidelines 
contained no requirement that peer review actually include pre-dissemination 
information quality review. Instead of providing a tool for preventing the 
dissemination of error, the guidelines made it possible for Federal agencies to 
use peer review to shield themselves from charges that information they 
disseminated is false. 

For these reasons, and others, I am skeptical of the BPC committee's 
recommendations for improving peer review. Some of them are unlikely to 
help, even if they could be implemented at no cost, because they are too 
ambiguous (e.g., "strengthen peer review," "experiment with different ways 
of conducting peer reviews") or contrary to self-interest (e.g., "[u]niversities 
should do more to make service as a peer reviewer an expected and 
appreciated aspect of a scientist's career"). For other recommendations (e.g., 
"[s]cientific journals should improve the quality control of peer review," or 
have "clear, publicly accessible conflict-of-interest policies"), the BPC 
committee neglected to offer strategies for actually implementing them. 

In my written testimony, I noted that much of the problem with 
governmental peer review is that the task relies on scientists but often 
involves substantial policy content. This policy content could be explicit (e.g., 
an agency seeks policy advice) or implicit (e.g., an agency seeks ratification 
of a risk assessment in which the agency's preferred policy is embedded in 
the methodology). I recommended that scientific peer reviews be strictly 
limited to science, noting several desirable attributes that would result. In 
my experience organizing strictly scientific peer review, it has been a 


Belzer (2002). 

Office of Management and Budget (2005). 
” Belzer (2012), pp. 21-25. 
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challenge to persuade scientists that policy matters are truly beyond the 
scope of the charge, but once they are convinced of this they find the peer 
review task much more interesting and intellectually stimulating. 

One of the challenges of implementing strictly scientific peer review is 
it requires a fundamental cultural change. This is generally not in the 
interests of Federal agencies that sponsor a peer review program they 
consider effective, so it will not occur systematically without congressional 
action. 

• "In presenting the conclusions of literature reviews, 

agencies and their scientific advisory committees need to be 
as open and precise as possible in discussing levels of risk 
and uncertainty. " 

This is part and parcel of the tyranny of single number risk 
characterizations, discussed above. What the BPC committee did not 
acknowledge, but what everyone knows, is that it often is contrary to an 
agency's actual or perceived interest to acknowledge uncertainty, much less 
give full attention to it. Agency risk assessors may do the right thing and try 
to provide full disclosure and analysis of uncertainty, but agency officials 
often do not want this information. They may find the information too 
complex or just psychologically unsettling. Further, agency attorneys tend to 
dislike disclosure of uncertainty because they fear that doing so compromises 
the defense of promulgated regulations. Courts are obliged by Chevron v, 
NRDC (467 U.S. 837, 1984) to give substantial deference to agency 
expertise, and deference is easier to give if the agency's experts say they are 
sure about something even when they have hardly any idea at all. 

Like a few other BPC committee recommendations, this one is mostly 
wishful thinking. Yes, it would be much better if agencies and their scientific 
advisory committees properly characterized variability and uncertainty when 
discussing risk. No, this is not going to happen unless and until Congress 
acts — ^to remove the ambiguity that creates agency discretion, to replace the 
Supreme Court's Chevron jurisprudence with something else, or to make the 
full disclosure or variability and uncertainty a nondiscretionary agency duty. 
To the extent that full disclosure is at least implicitly required by the Federal 
information quality standard of presentational objectivity, a simple remedy 
Congress can implement is to mandate that agencies adhere to this standard 
and make agency compliance judicially reviewable. 
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2 . This Subcommittee's oversight has highlighted a pattern in EPA 
science; the Agency has protocois or guidelines to encourage 
transparency, objectivity, or information quaiity, but these 
standards are often ignored. What steps couid be taken by 
Congress to ensure these standards are foiiowed? 

There are two general problems with the way agencies use guidance. 
First, they often publish it with the intent of achieving regulatory outcomes 
without bearing the burden of adhering to the rulemaking requirements of 
the Administrative Procedure Act. This practice is contrary to law but 
nonetheless widespread because of the savings to the agency if successful 
and the expense required to mount a legal challenge.^" Second, when 
agencies use guidance properly, such as to limit their own exercise of 
discretion in order to reduce uncertainty, they often refuse to honor these 
commitments. 

ERA'S information quality guidelines provide an excellent example of 
the latter phenomenon. These guidelines are generally well thought out, 
but EPA has not been forthright in honoring the commitments they contain. 
Similarly, ERA'S Peer Review Handbook mentions information quality but 
includes no provisions for actually integrating it into the peer review 
process. 

Several years ago, 0MB issued government-wide guidance on the use 
of guidance. In addition to a number of housekeeping provisions, 0MB 
established a pair of very simple and straightforward substantive principles: 

• Guidance must ”[n]ot include mandatory language such as 'shall,' 
'must,' 'required' or 'requirement,' unless the agency is using these 
words to describe a statutory or regulatory requirement, or the 
language is addressed to agency staff and will not foreclose agency 
consideration of positions advanced by affected private parties" 
(§II(2)(h));and 

• "Agency employees should not depart from significant guidance 
documents without appropriate justification and supervisory 
concurrence" (§ II(l)(b)). 


Congress could ameliorate this discrepancy by allowing plaintiffs who 
successfully challenge illegal guidance to recover their costs, perhaps including a 
penalty, from the agency's budget rather than the judgment fund. 

“ U.S. Environmental Protection Agency (2002). 

U.S. Environmental Protection Agency (2006). 

Office of Management and Budget (2007). 
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The first principle deals with the problem of using guidance illicitly for 
regulatory purposes. The second would require senior officials to explicitly 
waive limits on agency discretion contained in guidance and provide a 
justification for such waivers. In combination, these two principles reject the 
misuse of guidance for regulatory purposes and the failure to honor agency 
commitments. 

To make sure that EPA (or any other agency) complies with these 
principles, Congress would have to codify them in statute. 

3. EPA's recently-released final Scientific Integrity Policy 

"[ejstablishes the expectation that when communicating scientific 
findings. Agency employees include a clear explication of 
underlying assumptions, accurate contextualization of 
uncertainties, and a description of the probabilities associated 
with both optimistic and pessimistic projections, if applicable." In 
your view, has the Agency adequately followed this policy in the 
past? 

It is too soon for anyone to give an informed opinion concerning the 
extent to which EPA has followed this new policy, which has its genesis in a 
March 2009 Executive Order. My concern with the policy is I have mixed 
feelings about whether adherence to it is always beneficial. I have no qualms 
with the excerpt cited above, or with numerous other excerpts and 
provisions, especially those which promote greater transparency and 
objectivity. However, these excerpts are accompanied by other text that is 
problematic at best. 

A comprehensive scientific integrity policy must include provisions 
addressing both the politicization of science and the scientization of policy. 
After an extended delay, the Office of Science and Technology (OSTP) finally 
issued government-wide guidance in December 2010.^*" As my written 
testimony explained, this guidance handles the politicization of science 
ambiguously and the scientization of policy not at all.^^ 


Agencies try to comply with this second principle without ever tying their 
hands by including a standard disclaimer, such as the one included in EPA's 2010 
Scientific Integrity Policy. See U.S. Environmental Protection Agency (2010b). 

Obama {2009b). 

Holdren (2010). 

” Belzer (2012), pp. 2-3. 
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In some respects, EPA's scientific integrity policy is somewhat better 
than the OSTP guidance. For example, whereas OSTP prohibits only public 
affairs staff from altering scientific information, EPA's policy "Fplrohibits all 
EPA employees, including scientists, managers, and other Agency leadership, 
from suppressing, altering, or otherwise impeding the timely release of 
scientific findings or conclusions."^® 

While EPA says its policy "builds upon existing Agency and 
government-wide policies and guidance documents,"” its record of 
compliance with these other policies and guidance documents does not set 
an encouraging example. This concern is intensified by the disconnect 
between EPA's establishment of a cadre of "Scientific Integrity Officials" with 
responsibility to "champion" scientific integrity, "provide oversight for the 
implementation" of the policy, and "act as liaisons for their respective 
Programs and Regions," but no apparent authority for them to actually do 
anything. Months of work on the guidance did not enable EPA to eliminate 
useless, circular language.®® EPA says the policy repeats guidance issued by 
the Agency in 1999,®' so it’s not clear whether it contains anything new. In 
any case, EPA states that the policy is unenforceable by any entity other than 
EPA itself,®® so this may not be a distinction with any difference. 

In other respects, however, EPA's policy is much worse than the OSTP 
guidance. Like OSTP, EPA does not acknowledge the scientization of policy as 
a deficit in scientific integrity. Unlike OSTP, however, EPA's policy statement 
is not benign: it requires adherence to certain previously issued Agency 
guidelines whose very purpose is to scientize policy.®® Why include this 
particular reference in scientific integrity guidelines? It ensures that the 
scientization of policy is exempt, while simultaneously making it appear that 
Agency officials "politicize science" if they ever try to reclaim the authority 


U.S. Environmental Protection Agency (2010b), p. 4. 

Ibid. p. 1. 

®“ See, e.g. Ibid. p. 3 ("To support a culture of scientific integrity within the 
Agency, this policy ... [p]romotes a culture of scientific integrity"). 

Ibid. 

®® Ibid. p. 2; The Scientific Integrity Policy "does not create any obligation, 
right or benefit for any member of the public, substantive or procedural, enforceable 
by law or in equity by any party against the United States, its departments, 
agencies, or entities. Its officers, employees or agents, or any other person." 

®® Ibid. p. 4 (requiring adherence to EPA’s Guidance for Risk 
Characterization). 
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delegated by Congress to make policy decisions that are now made 
"scientifically" by career staff. 

Finally, EPA's scientific integrity policy includes an extensive section 
"[t]o assure the protection of Agency scientists," presumabiy from 
interference by Agency officials,^'' Notice that there is no parallel section 
protecting Agency officials from interference by Agency scientists in the 
exercise of their statutory authorities. 

4 . A recent joint report from the EPA's Science Advisory Board and 
Board of Scientific Counselors recommended that the Agency 
"include sustainability in its research vision" in order to allow 
"EPA to adopt sustainability as a core principle to inform decisions 
and actions. " Is this emphasis on sustainability appropriate for 
EPA’s research and science activities? Do you have any concerns 
about this new mission? 

A comprehensive review of the SAB and BOSC foray into 
"sustainability" must await a clear definition of the term. EPA's existing 
definition is highly subjective and too ambiguous to be measured. When a 
goal is subjectively defined or cant be measured, it can never be shown that 
it hasn't been achieved and it's anybody's guess whether achieving it is even 
a good thing. 

The joint SAB/BOSC letter has similar difficulties. It recommends that 
EPA's Office of Research and Development 

include sustainability explicitly in its research vision, invoke a definition 
of sustainability shared across ORD, and demonstrate clearly how 
planned research relates to the key components of sustainability (the 
environment, the economy, and society). 

In lieu of a coherent definition, SAB/BOSC point to a recent National 
Research Council report, which also lacks a clear definition. The NRC 
committee's review, which EPA sponsored, begins with numerous additional 
caveats. For example, the committee did not examine whether 


Ibid. p. 5; Sec. IV(A)(3). 

U.S. Environmental Protection Agency (n.d.). 

U.S. EPA Science Advisory Board (2011), p. 6. 

National Research Council (2011), p. 3. "'Sustainability' and 'sustainable' 
mean to create and maintain conditions, under which humans and nature can exist in 
productive harmony, that permit fulfilling the social, economic, and other 
requirements of present and future generations." 
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"sustainability," so defined, is consistent with the statutes that govern EPA's 
authorized activities.^® 

It is always a concern when a Federal agency embarks on a new 
mission that is not explicitly authorized by law, and even more so when 
cheered on by scientists whose advice is sought primarily because they 
happen to embrace that mission. Public choice theory predicts that agencies 
will do this to expand their authority, power, staff, and resources. This is 
particularly characteristic of agencies such as EPA that have largely achieved 
the statutory objectives Congress originally assigned to it. Like private firms, 
agencies strive to reinvent themselves, including the creation of new 
missions, when it becomes clear that the need for their goods or services has 
dwindled, or they have become irrelevant or overcome by technological 
change. Federal agencies differ, however, insofar as they have no 
constitutional role to engage in activities that Congress has not authorized. 

This suggests an inherent weakness in both the SAB/BOSC and NRC 
reports, and of course EPA's approach as well. Both reports could have, but 
did not, examine the extent to which the complicated and sometimes 
inconsistent patchwork of statutes that EPA implements has the perverse 
effect of making it harder for "humans and nature [to] exist in productive 
harmony." An obvious example might be the regulation of criteria air 
pollutants; the Clean Air Act can be interpreted to require the PM 2.5 and 
ozone NAAQS to be set at zero, in which case the statute is a suicide 
compact for humans and nature alike. 

5 . There was some discussion about the importance of objectivity 
and the roie of peer review in EPA risk assessments. 

a. Please describe how greater objectivity in assessments can 
be achieved, and what the practical effects of these 
improvements would be. 

Among the witnesses testifying before the Committee on February 3, 
there appeared to be universal agreement that objectivity is not optional in 
science. Objectivity is an essential attribute of the scientific method, one that 


Ibid. pp. 17-18. The NRC committee attempts to show that Congress 
authorized EPA to implement sustainability via the National Environmental Policy Act 
of 1969. This logic is circular. NEPA did not establish "sustainability" as a governing 
principle; rather, President Obama borrowed hortatory language from NEPA to define 
"sustainability," whose definition the NRC committee then used. See Obama (2009a), 
p. 52126. Sec. 19(1). 
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is required via government-wide and EPA information quality guidelines.^® 
Sometimes, objectivity (or one of its synonyms, "accuracy") is explicitly 
required by law.'*® 

The hearing did not effectively distinguish between "science" and "risk 
assessment," however. EPA risk assessments are routinely described as 
scientific, even by Agency officials. This Is not correct. EPA risk 
assessments are notoriously lacking in objectivity, and as I testified before 
the Committee, this is a matter of design, not accident.'*^ 

Objectivity in risk assessment would serve three crucial purposes that 
current practices do not. First, it would ensure that EPA officials. Congress, 
and the public had unbiased information about the risks within EPA's 
jurisdiction. Second, it would enable these risks to be ranked so that 
resources devoted to risk reduction could be rationally allocated. Third, the 
authority to make risk management decisions that Congress has delegated to 
Agency officials would finally be made by those Agency officials, not by 
Agency scientists and career program managers with strong policy views. 

b. Please describe the different types of peer reviews that 
science funded by or used by EPA may be subjected to, 
including which areas raise the most concerns. 

As I have noted elsewhere in these replies and in my written 
testimony, peer review takes several different forms. Moreover, the purpose 
of scholarly peer review (to ration scarce journal pages) is fundamentally 
different than the purpose of governmental peer review (to ascertain what is 
correct). The procedures used for the former are ill-suited for the latter. For 
this reason, too much emphasis has been placed on scholarly peer review. 

EPA peer review is governed by 0MB guidelines and EPA’s Peer Review 
Handbook, which have useful features but serious limitations and defects, as 
I have already discussed. To recap, these include too much Agency influence 


Office of Management and Budget (2002), U.S. Environmental Protection 
Agency (2002). 

See, e.g.. Clean Air Act, Sec. 108(a)(2): "Air quality criteria for an air 
pollutant shall accurately reflect the latest scientific knowledge useful in Indicating 
the kind and extent of all identifiable effects on public health or welfare which may 
be expected from the presence of such pollutant in the ambient air, in varying 
quantities" (emphasis added). 

See, e.g., Anastas (2011), oral testimony. 

Belzer (2012), p, 5. 




173 


Committee on Science, Space, and Technology 

"Quality Science at EPA: Perspectives on Common Sense Reform- Day 11" 
Responses to Questions for the Record 
Page 19 


over the selection of reviewers and absolute control over their charge. EPA's 
peer review process is designed to maximize concordance with EPA's policy 
objectives. Einally, peer reviews of EPA risk assessments are suffused with 
risk management poiicy judgments. 

In my testimony, I recommended that EPA peer reviews be strictiy 
limited to science. I aiso suggested other reforms to the process, such as 
giving the most knowledgeable researchers on a scientific issue the 
responsibility of educating peer reviewers and coordinating open debate in 
which the public could actively participate. Unlike EPA practice, peer 
reviewers would never be drawn from the ranks of researchers who have 
published research or taken positions on the specific issue. 


QUESTIONS FROM REP. RANDY NEUGEBAUER 


1. What type of research does the EPA currently conduct to confirm 
predicted health outcomes of previously promulgated rules? In 
what ways can we improve existing efforts to examine the real 
effects of EPA rules, and what else can the Agency be doing to 
increase accountability for health and environmental predictions? 

I am not aware of any significant EPA effort to confirm predicted 
health outcomes for previously promulgated rules. The closest thing I can 
think of is EPA's reconstructions of the benefits and costs of the Clean Air 
Act, as it was required to do under Section 812.'’^ These reports do not 
confirm anything, except perhaps the foolishness of asking an agency to 
conduct its own performance evaluation. 

If Congress is serious about estimating how actual health outcomes 
compare with predictions, it must ensure that the review is conducted 
rigorously, independently, and transparently. Not only does this exclude EPA 
from performing the review, it also excludes the National Academy of 


Se_e most recently, U.S. Environmental Protection Agency (2011). 

For a review of EPA's first foray into self-examination in this area, see 
Lutter and Belzer (2000). 
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Sciences, which does not practice transparency in the selection of experts, 
committee deliberation,'*® or internal peer review.'*'' 

2 . Are there any legitimate concerns about making public the data 
sets that attempt to Justify EPA reguiations? Are there real 
concerns about confidentiality, and if so, do they outweigh the 
necessity of accountabiiity, transparency, and open and sound 
scientific examination? 

There are only two areas in which confidentiality is a legitimate 
concern in human health risk assessment. The first involves confidential 
business information, such as perhaps studies performed on proprietary 
mixtures. I am unaware of any serious controversies in this area, but they 
may exist. 

The second involves personally identifiable information, which 
epidemiologists long ago learned how to anonymize. It is an exceedingly rare 
study in which the identity of subjects must be known by the researchers and 
statisticians analyzing the data. Usually, knowledge of subjects' identities, 
whether they belong to a case or control group, and similar matters are 
purposely hidden from researchers themselves to ensure objectivity. 

Researchers often desire not to disclose their data because they 
consider it their own intellectual property. The case for this is very weak 
when the data collection was publicly funded, but strong otherwise. In my 
testimony, I offered a straightforward solution: EPA should be required to 
fully disclose any data it intends to rely upon for risk assessment or any 
component thereof. Disclosure of federally-funded research, to which the 
government already has a right, would be mandatory if the study met this 
condition. Similarly, any third party that wants EPA to rely upon its data 
would have to meet the same disclosure standard. 

My approach would impose no involuntary burden on researchers, 
federal or otherwise. Researchers could decide whether to restrict access to 
their data or influence public policy, but they no longer would be allowed to 


'*^ The National Academies (2003), The National Academies (2005), p. 6. 

The National Academies (2005), p. 10. "Committee meetings, particularly 
as the committee gathers information, are usually open to interested individuals and 
the news media. However, meetings are closed when the committee is deliberating 
to develop its findings and during discussion of financial and personnel matters. 
Closed meetings are not open to the public or to any person who is not a 
committee member or an official, agent, or employee of the Academies." 

'*■' The National Academies (2008). 
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do both. Full disclosure must be the price for seeking to influence public 
policy, regardless of the source of funding. 


QUESTIONS FROM REP. PAUL TONKO 


1. Private Consulting. 

Prior to the February 3rd EPA E&E Subcommittee hearing. 
Ranking Member Mr. Miller asked you to provide a list of the clients 
for whom you conduct private consulting work that may have an 
interest in the subject matter of that hearing. You declined to 
identify any of your past or present clients because you claimed a 
"confidentiality agreement" exists between you and your clients and 
that providing this basic information would violate that agreement. It 
is my understanding that the majority of "confidentiality 
agreements" revolve around the specific issues that a 'client' hires a 
'contractor' to perform, particularly the results of that work. It is less 
common that a "confidentiality agreement" would bar from public 
disclosure the mere fact that a business relationship exists between 
a client and a contractor. 

• Please list all clients you have signed a confidentiality 
agreement with broadly related to the subject of environmental 
science and/or regulatory issues over the past five years. 

The public disclosure of the existence of a confidential relationship 
would be tantamount to breaching its contents. As I said in my reply to 
Ranking Member Miller, I intend to honor these agreements and thus 
respectfully decline to identify past and present clients. 

It is clear from information already in the public domain that 
you worked for the U.S. Department of Defense, through a consulting 
agreement with Booz Allen Hamilton; advising DOD on issues 
revolving around the chemical Perchlorate in 2003 and 2004. 

• Please indicate if you still have a business relationship with Booz Allen 
Hamilton and/or DOD? If not, please indicate when that relationship 
ended? 

I do not have a current business relationship with Booz Allen Hamilton 
and/or DoD. I believe that my previous relationship ended in April 2005. 
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2. Public Disclosure. 

At the Feb. 3rd hearing you said: "When I do things that are 
public, when I put my name on something, a piece of work that I've 
produced then I disclose who I did it for. " It was unclear from your 
response, however, what work products you consider to be "public". 

• Please provide a list of work products you have produced over 
the past five years where you publicly disclosed who you did 
this work for, as you indicated you had done in your 
congressionai testimony. Please include the title or name of the 
work product, the client's name you performed the work for, 
and where and when this work was presented or appeared. 

I have uploaded to my personal website at ymyy,rbbelz^,cpm every 
publication, presentation, public comment, or similar work product that I 
have been able to locate — peer-reviewed or otherwise. For work products 
that are covered by a copyright owned by someone else, I provide links to 
web sites where copies can be purchased. I also have disclosed documents I 
authored on behalf of the Federal government. 

Some of this work pre-dates the Internet era, however, so my website 
is regrettably incomplete. 
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LI.S. HOUSE OF REPRESENTATIVES 
COMMITTEE ON SCIENCE, SPACE, AND TECHNOLOGY 
Subcommittee on Energy & Environment 

Hearing Questions for the Record 
The Honorable Andy Harris 

Fostering Quality Science at EPA: The Need for Common Sense Reform Day II 
February 3, 2012 

Dr. Jerald Schnoor 

1. Tlie Bipartisan Policy Center’s 2009 report, “Improving the Use of Science in Regulatory 
Policy” was mentioned during the hearing. This report made several suggestions that may 
be useful in guiding this Subcommittee’s efforts to reform regulatory science, including: 

• “Studies used in the formulation ofregulalion should be subject to data access 
requirements equivalent to those under the Data Access Act.” 

• “The process of conducing literature reviews” and “the process of naming advisory 
committees” should be made more transparent. 

• “Agencies should avoid turning repeatedly to the same scientists for service on 
advisory committees.” 

• Executive branch agencies need to “help clarify for both officials and the general 
public which aspects of disputes are truly about scientific results and which concern 
policy,” 

• “Policy makers should be wary of conclusions of risk that are expressed as a single 
number.” 

• “Federal agencies need to experiment with ways to increase the number of scientists 
who participate in peer review.” 

• “In presenting the conclusions of literature reviews, agencies and their scientific 
advisory committees need to be as open and precise as possible in discussing levels 
ofrisk and uncertainty.” 

Do you agree with any or all of these recommendations? Do you have any additional 
comments or advice in pursuing these goals? 


! went to ilic Bipartisan Policy Center’s website and found that this 2009 report was listed 
as an “interim report”. I could not download the report from their site, so i was not able 
lo read about these recommendations. It is difficult to coniinetil without having the 
context oflhe recommendation. However, 1 would stale that during iny pas! service to 
EPA Board of Scientific Counselors, EPA Science Advisory Board, NIEIIS Advisory 
CoiinciL and National Research Coundl/Nalional Academies ofScience, I have not 
become aware of problems associated with these points. 1 believe the process of naming 
advisory committees is quite transparent, that advisory committee members are regularly 
rotated, that most reports try to include an estimate of uncertainty, and that the Federal 
Advisory Committees Act (FACIA) requires openness (by law). Regarding the di.stinclion 
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belweeii science and policy recommendations, it is usiudiy clear whether the committee is 
commenting on science or poiicy. Many limes, advisory committees arc spccificaliy 
asked to critique or evaluate policy. In addition, research in the social sciences 
('iocial/bchavioral/decision sciences, including economics) often comprises ‘'policy 
rcscarcifh, and the dislinclion is sometimes blurred. 


2. In discussing problems with the EPA Science Advisor}^ Board process, Mr. Walls noted 
that “qualified scientists from industry should be given equal consideration” to serve on 
advisory panels. Do you support that recommendation? 

I do not support quotas, but i do support inclusion of members from industry along with 
oilier groups, I think EPA tries to attract qiialifcd scienlisls Iroin industry for the SAB and 
other Boards, but somelinies it is difncidl for these scienlisls to afford die lime to serve. It 
would be desirable to have more scientists from industry, I ihink. 

.4 good advisory comniittec should include members w’ho have distinguished themselves as 
leaders in the field by virtue of their publications, citations, patents, arvards and 
recognilion. They should embody a diversity oi' backgrounds, affiliaticms, expertise, 
experience, and view-points. Industry scieritisis should certainly be among the experts 
considered and selected. 1 have not served on an advisory coiiiniiltee in iny memory w'hich 
did not include merabers from industry. Experts from NGOs, acadeniia, business, industry, 
and government should all be considered. 


3. Many EPA science activities are housed within regulatory offices. For example, nearly 
half of EPA’s laboratories do not reside in the Office of Research and Development. In 
your view, should scienlitlc activities be organizationally insulated from regulatory 
activities to ensure objectivity and balance? 

No, I do not believe that one should “organizationally insulate” scientific from regulatory 
activities in EPA, On the contrary, I would like to sec better coordination of science to 
achieve the policy mission of EPA throughout the Agency. Better coordination is needed 
lo leverage scientific research ainorig ORD, Agency Offices (Water, Air, Radiation, etc,). 
Regional Offices, and the Siale.s/Tribes/Naiions. The mission of EPA is to proleci huinan 
health and the envirorirneni from pollution and other agenls. Quality science is needed to 
inform the policy decisions within the Agency for that protection at all levels. 


4. In an August 2010 editorial in the journal Environmental Science and Technology titled 
“Regulate, Baby, Regulate”, you asserted that hydraulic fracturing associated with shale 
gas development was “causing tap w'atcr to bum.” Please provide evidence lo support this 
assertion and describe how they relate to the standards employed by the publication in 
ensuring accuracy and appropriate communication of scientific uncertainties in journal 
editorials. 


In the editorial, I was referring to the dociiraentary Gasiand regarding the phrase “causing 
tap water to burn”. But, indeed, the problem may emanate from improperly constmeted or 
maintained wells on the pan ofbolh homeowner and indu.slry. Osborn ei al. (20! 1) in ilic 
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Froceedings of the National Academy of Sciences (doi : 1 073/pnas. 1 1 00682 ! 08) article, 
"Methane coniaminalion ofdrinkiiig water aceornpanying gas-wcIi drilling and hydraulic 
fracturing”, found that 5 i ol’dO homeowner wells wilhin one kiiorneler oi'new natural gas 
weds were contaminated wiih deep methane of shale origin in the Marceilus and Ulica 
shale formations of noitheasTeni Pennsylvania and upstate New York. Corresponding 
author. Robert B. Jackson,, Duke University, said, “M saw a honicovvncr light his water on 
fire " the biggest risk is flammability and explosion" 

( http:/’/'w\vw’.sciciiti ticaincri can. com, -''a'itidc.cfmlhd-^^-hrackmg-for-naliiraj-gas-pol lutes- 
's.^'aier- we j I -s&pr i n l'~^lrue ) . 

The PNAS article is by no means the last data on the subject. Ceitainly, there is more to 
icam about how contamination of wells may occur. And there arc foilow-up studies being 
undertaken, including one by EPA. But it is clear that gas weJis must be properly 
constructed, mainteiiicd, and plugged after operation. Such diligence will require careful 
regulation on the part of EPA. Fracking chemicals should be made known to EPA; the 
pLibhcN safety over-rides the need lor proprietary privilege here. Furlliermorc, wells 
shoiiid be plugged and mdioaclive conlaniinanls m pits, ponds and lagoons from fracking 
operation return flovys must be treated or otherwise reinjected. 
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L1.S. HOUSE OF REPRESENTATIVES 
COMMITTEE ON SCIENCE, SPACE, AND TECHNOLOGY 
Subcommittee on Energy & Environment 

Hearing Questions for the Record 
The Honorable Randy Neugebauer 

Fostering Quality Science at EPA: The Need for Common Sense Reform Day 11 
February 3, 2012 

Dr. Jerald Schnoor 

1. What type of research docs the EPA currently conduct to confirm predicted health 

outcomes of previously promulgated rules? In what ways can w^e improve existing efforts 
to examine the real effects of EPA rules, and what else can the Agency be doing to increase 
aceoimtahility for health and environmental predictions? 


Further studies are needed to conilrm ibat human heallli and ecosystems improve fojlowing 
new regulations - I will call such studies ''posl-audil's'’. But these post-audits will be 
expensive and funding will be required, in ray wxitten testimony delivered on I'ebruary 3, 
2012, 1 mentioned an example offoUow-up studies that were made to the Harvard Six 
Cities epidemiologic report by Dockery ct al (1993), which first implicated the role of line 
particles (PM 2 . 5 ) in morbidity and mortality from air poiiiition. There remained questions 
about, the etiology of the disease and the surprising results of cardiopulmonary deaths, so 
foHovv-up studies (post-audits) were underlaken. In 2000, a reanaiysis ofthe Harvard Six 
Cities Study was reported by the Health Effecls Inslitute (HFI) and. in 2004, research was 
completed that validated the initial results (Krew'ski et at., 2004). An extended follow-up 
study by Francine Laden and colleagues was subsequently published in 2006 (Laden et al, 
2006) as well as a summary ofthe beneficial effects on life expectancy by Pope ct al. 
(2009). Laden was quoted in a Harvard School of Public Health Press .Release at the time: 

“Thejbihw-up study found that an average ofdv'ee percent fewer people died for every 
reduction of one microgram per cubic merer in the average levels ofPM2.5fme particulate 
matter, defined as having a diameter of 2.5 microm or less - narrower than the width of a 
human hair. This decreased death rate is approximate to saving 7.5, 000 people per year in 
the U.S. ” 

Tlial's an example of' the scienlilic process performing well. A bypolliesis was tested, 
results obtained, questions arose, and furthcT studies were underlaken lo confirm the 
results. Continual challenges to a .standing theory or hypothesis are recorded in the 
scienlilic literature. Each research article must pass rigoroius peer review, and subsequent 
studies either confirm the initial results or posit a new theory. The theory stands until 
future studies disprove .it (if ever). 1’hat is how the body of knowledge grows. 
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2. Are there any legitimate concerns about making public the data sets that attempt to justify 
EPA regulations? Are there real concerns about confidentiality, and if so, do they 
outweigh the necessity of accountability, transparency, and open and sound scientific 
examination? 


Transparency of scientific results senses the public’s best interests. However, sometimes it 
is difficuli or even impossible for an incltvidiial scieruisi lo respond lo all the requests that 
s/he may receive. They may not have the resources or time required lo respond. RiCu 
regardless, their data should be made available to the public through the journal in which it 
was published. Other investigators need the original data because they may attempt to 
reproduce the study. Of course, the names and identities of individuals in the data set may 
be confidcniia! as required by federal low in the case of hcailh studies and personal 
information (HIPAA, i'ERPA). 
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t.S. HOUSE OF REPRESENTATIVES 
COMMITTEE ON SCIENCE, SPACE, AND TECHNOLOGY 
Subcomnilttec on Energy & Environment 

Hearing Questions for the Record 
The Honorable Brad Miller 

Fostering Quality Science at EPA: The Need for Common Sense Reform Day U 
Eebmary 3, 2012 

Dr. Jerald Schnoor 


EPA ORD Realignment 

1 . In the realignment of the Office of Research and Development programs you both 

described in your testimony, has the reduction and integration of the number of programs 
down from 1 3 to 6, proven to be a more effieient and effective strategy for EPA-ORD 
research aelivities? How has this alignment improved ORD’s coordination with the 
program and regional offices? What more can and needs to be done to strengthen the EPA 
overall research enterprise? 

It is a bit early to determine if the realignment of 'EPA ORD programs has been successful 
because it is just being itndenakcn in 2012. However, 1 believe the restracturing makes 
sense — it was in concert with advice provided in recent years by both the Science 
Advisory Board. (S.AB) and the .Board of Scientific Counselors (BOSC). Thirteen major 
programs proved somewhat unwieldy, and the proposed realignment received positive 
review fro,ni the SAB ($,AB, 201 1). Molivalion for Lius consolidation and reaLigmnenl of 
programs reflects an emphasis on in,teg.rated trans-discipiinary researc.h, multi-pollutant 
exposures, and sustainability. These are not .new programs, but represent a new way of 
thinking within OK..D. Considerable synergies may be realized in coinbiiiiitg research into 
the four programmatic areas: Air, Climate and Energy; Safe and Sustainable Water 
Resources (water quality plus drinking water); Siistainable and Healthy Conimimilicfi; and 
Chemical Safely for Sustainability; plus two smaller programs in Homeland Security 
Research and Human Heallh Risk Assessment. ORD is evolving from a risk management 
paradigm, which has guided and inlluenced research over the past two decades, towards a 
sustainability paradigm. That effort will likely pay dividends. It is consistent with a public 
health. approa.ch of '‘preventing disease"' rather than a medical approach to “treaung 
disease'” after li occurs, and it recognizes that environment and health are an interconnected 
system. It also follows from early pioneering research at EPA on Pollution Prevention at 
the source, rather than cnd-oi-pipc. 
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U.S. HOUSE OF REPRESENTATIVES 

COMMITTEE ON SCIENCE, SPACE, AND TECHNOLOGY 
Subcommittee on Energy & Environment 

Hearing Questions for the Record 
. TTie Honorable Andy Harris 

Fostering Quality Sei^ce ot EPA: The Need for Carnmon Sense Reform Day II 
February 3, 2012 

Dr. S. Stanley Young 

1. The Bipardsan Policy Center's 2009 report, "Improving the Use of Science in Regulatory 
Policy" was mentioned during the hearing, This report made several suggestions that may he 
usefiil in guiding this Subconunittee's efforts to reform regulatory science,. including: 

• "Studies used in the formulation of regulation should be ^bject to data access 

requirements equivalent to those under the Data Access Act" 

-• "The process of conducting literature reviews" and '‘the process of naming advisory 

committees" should be made more transparent 

• "Agencies shodd avoid turning repeatedly to the same scientists for service on advisory 
committees." 

• Executive branch agencies need to "help clarify for both officials and the general public 
which aspects of disputes are truly about scientific results and which concern policy." 

• "Policy makers should be wary of conclusions of risk that are expressed as a single 
number." 

• "Federal agencies need to expaiment with ways to increase the number of scientists 
who participate in peer review." 

• "In presenting the conclusions of literature reviews, agencies aid their scientific 

• advisory committees need to be as open and precise as possible ia.discussing levels of 
risk and uncectaiiity." 

Do you agree with any or all of ihese recommendations? Do you have any additional 
. comments or advice in pursuing these goals? 

2. In discussing problems with the EPA Science Advisory Board process, Mr. Walls noted that 
“qualified scientists from industry diould be given equal consideration” to serve on advisory 

' panels. Do you support that recommradation? 

3. Dr. Schnoor’s testimony cites work done on the Huk between fine particulate matter (PM) 
and premature mortality by Dr. Fraricine Laden, induding the claim that PM-related 
regulations had saved more than 75,000 lives, as “an example of quality science performing 
well”. Have you examined this literature based on epidemiological studies, and, if so, do you 

find it rigorous enough to justify’ major, costly EPA regulations? 
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4.' Snme people 'have expressed' coBcems about the practical effects of mald-'Qg-iJiiderl3hng data' • 
sets publicady available, cMng significant costs aad concerns about oonfidcntiality. Are -diese 
concerns valid? Why or -wliy not? To the extent you believe such concerns are valid, how 
might they be addressed to miniipize potential negative impacts ofmaking data publicly 
a'railable? 
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D.S. HOUSE OF REPEESENTATTOES 
COMMITTEE ON SCIENCE, SPACE, AND TECHNOLOGY 
Snbcommittee on Enei^ & Environment 

Heaiii^ Questions for the Record 
The Honorable Randy Neiigebaner 

Fostering Quality Science atEPA: The Need for Common Serae Reform Dc^II 
February 3, 2012 

Dr. S-. Stanley Young 

1 . WiLat ofresearoh does the EPA currently conduct to confirm predicted health - 
outcomes of previously promulgated rules? In what ways can we improve existing 
eferts to examine the real effects of EPA rules, and -v^at else can the Agency be doing to 
increase accountability for health and environmental predictions? 

2. Are diere any legitimate concerns about malnrig public the data sets that attempt to justify 
EPA regulations? Are there real concerns about confidentiality, and if so, do they 
outweigh the necessity of accountability, transparency; and open and sound .scientific 
examination? 
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Responses by Dr. S. Stanley Young, 

Assistant Director for Bioinformatics, 

National Institute of Statistical Sciences 

Questions Dr, Young 

In general I fully support the statements in Question 1 . What follows are specific 
comments to parts of that question. 

Bullet #3. “Agencies .should avoid turning repeatedly to the same scientists. , 

I would strengthen this in the following w'ay. Scientists should serve for no longer than 3- 
years tenns and not be eligible for re-appointment for 3 years after serving. If service is 
expected to be a public service, it should not be turned into a method to secure 
government grants. 

Congressman .loe Barton commented: “Almost every single member of your Clean Air 
Science Advisory Committee (CASAC) has been directly or indirectly funded for 
research.” According to Steve Milloy, examination of the EPA extramural grants data 
base, found that members had research grants from the EPA: 

• CASAC chairman Jonathan Samet is listed a principal investigator on 
$9,526 . 921 in EPA grants. 

• Board member George Allen received is listed as a principal investigator on 
$3.907.111 in EPA grants. 

• Board member Ana Dlez-Roux is listed as a principal investigator on 
$31.343.081 in EPA grants. 

• Board member H. Christopher Frey is listed as a principal investigator on 
$2,956,342 in EPA grant.s. 

• Board member Armistead Russell is listed as a principal investigator on 
$2 0,13 0,736 in EPA grants. 

• Board member Helen Suh is li.sted as a principal investigator on Si 0.962,364 in 
EPA grants. 

• Board member Kathleen Weathers is not listed as a principal investigator on any 
EPA grants; but her employer, the Cary institute of Ecosystem Studies, is listed as 
a the lead institution in S3. 570,926 in EPA grants. 


Bullett#5 “Policy makers should be wary...” 

It is a problem if policy is decided and then supporting science is sought. It is not just a 
summary into one number, which is a problem. Any justification of a regulation should 
list the science papers that support a policy and those that oppose the policy. 

A single number can be misleading. For example, a cost benefit ratio can be 5 where we 
have S5/$l=5 (trivial), or $5B/S1B (having major impact). Papers justifying both 
numerator and denominator should be provided. 

Bullett#6 “Federal agencies should experiment ...” 

Peer review of agency science or actions is not enough to ensure validity of agency work. 
One way to expand oversight is to require that data used in papers that the agency is 
relying on is public. Specifically, the agency should identify important papers, papers 
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being used to help justify policy decisions, data used in these papers should be secured, 
electronic (easily readable and ready for analysis) copies of this data should be posted on 
the agency web site. 

Iftbe agency is funding work, then it should seriously consider funding data collection, 
staging and posting separately from data analysis. The group collecting the data should in 
no sense “own” the data. The data should be a public good. 

If the data is public, then the w'hole scientific process can be used for peer review. 

Question 2. “...qualified scientists from industry...” 

Scientists do not live in a financial vacuum. University scientists compete for grants and 
industrial scientists work to advance their company. Hopefully data and methods will 
lead both to sound conclusions. If government funded scientists are allowed to 
participate, so should industrial scientists. 

Questions 3. “... link between ...PM and premature mortality...” 

I have examined this literature to some degree. For acute death, s, if a person is going to 
die within two weeks and they die .sooner by a few days rather than later in a few weeks 
has a life been saved? Most of the literature I have read is not capable of addressing this 
issue. For long-term deaths, environmental epidemiology is not capable of reliably 
detecting effects as small as those observed. The risk ratios at issue, less than 1.5 and 
often as small as 1.05, could easily be the result of some bias in the data. 


Two recent papers. Laden et al. (2007) and Birdsey et al. (201 1), of diesel give 
conflicting claims. Neither paper obsen'es an increase in all-cause mortality due to diesel 
exposure; both appeal to the “healthy worker effect”. Laden et al. (2007) claim an 
increase in lung cancer, but the confidence limits overlap 1 .00 so most would not 
consider their evidence compelling. There is also a claim an increased heart disease with 
a RR of 1 .41 . Birdsey et al. (201 1 ) find neither an increase in lung cancer nor any effect 
on heart disease. 

Birdsey et al, comment: “Previous research suggests that truck drivers are at increased 
risk for lung cancer ..., prostate cancer ... heart disease, hypertension..., stomach ulcers 
..., bladder cancer, and stomach cancer.” 

but find with their data set that 

“Mortality was not increased for any of the health conditions previously shown to be 
elevated among truck drivers ..., although the lack of statistically significant estimates 
for cancers of the stomach, bladder, and prostate indicates that e.xcess mortality from 
these causes cannot be totally eliminated (Table 3).” Note that Laden et al. do not 
confirm previous claims. Bias, multiple testing and multiple modeling are likely all at 
play, and as Koop et al, cited later, indicate, findings from observational environmental 
studies can be unreliable. 
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Some, e.g. Laden, make a claim for a very small effect and then multiply their small risk 
times the US population to make a claim for a large number of statistical deaths. Claims 
coming from epidemiology studies are notoriously unreliable. Young and Karr (2001). In 
the case of the Laden claims again.st die.sel are not supported by Birdsay. Neither Laden 
nor Birdsay lind any effect on mortality for truck drivers. 

A recent paper where Laden is a co-authors states, “Conclusions: Among this cohort of 
men with high socioeconomic status living in the midwestern and northeastern United 
States, the results did not support an association of chronic PM exposures with all-cause 
mortality and cardiovascular outcomes in models with time- varying covariates.” So 
Laden appears to be on both sides oflhe air pollution and mortality question. 

A recent press release from the EPA made a claim of 1 60,000 lives saved in 20 1 0. Again, 
very small and quite variable risk estimates arc multiplied by the size of the US 
population. The risk estimates are all well within the range of study bias. The press 
release also makes a claim for preventing more than 170,000 asthma attacks. The Koop et 
al.(20 1 0) and therefore calls those claims into question as they saw' no increase in 
hospital admissions for asthma related to air pollution. 

Birdsey J, Altcrman T, Li J, Petersen MR, Scstito J. Mortality among members of a truck 
driver trade association. AAOHN Journal 2010. 58, 473-480. 

EPA Press Release 

hUp://yosemite.epa.gov7opa/admpress.nsf/7ebdr4d0b217978b852573590040443a/r8ad34 

85e788be5a8525784600540649!OpenDocument 

Koop G, Ross McKitrick R, Tole L. Air pollution, economic activity and respiratory 
illness: Evidence from Canadian cities, 1974-1994. Environmental Modelling & 

Software 2010 25, 873-885. 

Laden F, Hart IE, Smith TJ, Davis ME, Garshick E. (2007). Cause-specific mortality in 
the unionized US trucking industry. Environmental Health Perspectives. 2007. 115, 
1192-1196. 

Puett RC, Hart IE, Suh H, Mittleman M, Laden F. (2011) Particulate matter exposures, 
mortality, and cardiovascular disease in the Health Professionals Follow-up Study. 
Environ Health Perspect 1 19:1 130-1 135. 

Young SS, Karr A. Deming, data and observational studies, a process out of control and 
needing fixing. Significance. September201 1, 1 16-120. 

Question 4. Cost of making data available 

Data is collected, then staged into an “analytical data set” suitable for statistical analysis. 
The collection and staging of data is expensive, but it has to be done for any published 
paper. The cost of making this analytical data set electronically available is typically 
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quite low. With planning, meta data describing the analytical data set is also inexpensive 
to produce. One to a few sentences can be given to instruct another analyst what the 
variables mean. There is a NSF funded data repository, see datadryad.org, where the 
analytical tile and the meta data can be deposited, which removes the burden or 
responding to individual data requests. In short, once data is staged to be used in a 
publication, the incremental cost of making data available is low. 


General Comments; 

Question from Chairman Harris, “What is the EPA doing in their R&D budget to look at 
w'hethcr or not these health benefits that arc claimed actually come to pass in the 
magnitude that they are claimed?” 

Answer: One of the supposed strengths of science is that over time replication of results 
gives results credence. There is a premium in science for being first with a claim and also 
journals favor nevv results, not replication of results. The agency could and should fund 
studies attempting to replicate important claims. In the area of medical observational 
studies, claims most often fail to replicate. See Young and Karr (201 1). loannidis, JAMA, 
(2005) notes that 5/6 claims coming from medical observational studies failed to replicate 
or the claimed si/e ofthe elTect was greatly reduced. The agency should identify 
important claims and fund independent studies to determine if the claims replicate. For 
example, do claimed mortality affects of PM2.5 in the East replicate in the West? Do 
claimed effects during the time 1990-1999 replicate during the time 2000-2009? Note 
that making data available at the time of publication of a paper would greatly speed up 
the scientific evaluation of the claims made in the paper. 

Mr. Greenbaum notes that scientists within the agency are both “creating some science 
but they are then involved in the interpretation ofthat science.” A clear separation of data 
collection and staging from data analysis and interpretation would help the scientific 
process. For example in a FDA supervised clinical trial, the statistical analysis is pre- 
spccified. Data is collected and staged. Finally the cleaned data is given to statisticians 
for analysis. The data is also given to the FDA. No one group is responsible for all steps 
in the process. loannidis, JAMA, 2005, notes that claims coming from randomized 
clinical trials replicate over 80% of the time. Administrative procedures can be put in 
place to support normal science. Make data sets publicly available on the publication of a 
paper. To verify important claims, independent studies should be funded. Separate data 
collection and staging from analysis and interpretation. Etc. 

Correction: Line 1012. Change “data.” to “paper.” 

Line 1037 and again Line 1373. Mr. Schnoor makes the statement, “As a journal editor, 
our policy is, and most journals, that if the data is asked for, we do give it.” It is a fact 
that most journals do not require that authors make their data available as a condition of 
publication. Science, Nature, PNAS are exceptions. Even when authors sign agreements 
to make data available, most often they do not. About 2/3s of the time, data asked for is 
not provided. One solution would bo to require the data files used in analysis be deposited 
with the journal or a data repository, datadryad.org. 
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Line I204ff. “Is the science conducted, in your opinion, by the EPA generally of poor 
quality?” 

Answer: We really do not know the quality of their work for several reasons: 

1 . Strictly internal work is not subjeet to independent peer review. 

2. Papers they publish are subjected to peer review, but peer review most typically 
does not examine the data set and reanalyze it. Peer review only says that paper 
meets the generai standards of the area. Note well: for observational studies 
claims most often do not replicate. Young and Karr, 2011. 

3. Much of the critical work of the EPA is via contract. EPA scientists do not 
nece.ssarily get the data sets (as the FDA would) and they do not do an 
independent re-analysis of the data. 

We ai'e largely observing “trust me” science. At best we have no idea of the quality of 
their work. At worst, claims coming from environmental epidemiology studies are no 
better than claims coming from medical observational studies. With medical 
observ'ational studies claims fail to replicate 80 to 90% of the time, loannidis, JAMA, 
2005 and Young and Karr (201 1). 

Line 1442 Mr. Greenbaum, “...we cannot have detailed re-analysis of every .single 
paper...” 

Answer: If the data is publicly available, then anyone w'ith an interest can have a re- 
analysis computed. Support the normal working of science by making data public. For 
key claims, the agency should fund an independent rc-analysis or indeed a new study. 
Remember, we are not working in an area where claims are generally reliable. 
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The replication of important findings by multiple independent investigators is fundamental to the accumulation of I 

scientific evidence. Researchers in the biologic and physical sciences expect results to be replicated by indepen- § 

dent data, analytical methods, laboratories, and instruments. Epidemiologic studies are commonly used to quantify g_ 

small health effects of important, but subtle, risk factors, and replication is of critical importance where results can 5 * 

inform substantial policy decisions. However, because of the time, expense, and opportunism of many current 3 

epidemiologic studies, it is often impossible to fully replicate their findings. An attainable minimum standard is •§• 

“reproducibility," which calls for data sets and software to be made available for verifying published findings and 
conducting alternative analyses. The authors outiine a standard for reproducibility and evaluate the reproducibility 
of current epidemiologic research, They also propose methods for reproducible research and implement them by o' 

use of a case study in air pollution and health. 3 


air pollution: information dissemination; models, statistical 


Abbreviation: NMMAPS, National Morbidity, Mortality, and Air Pollution Study. 


Deterniinaats of tiuinaii disease are coinmuniy investi- 
gated by epidemiologic studies focused on a particular sub- 
population, time frame, and geographic location. Findings 
from such studies can play an important role in policy deci- 
sions affecting public health (1). Yet epidemiologic research 
has been criticized as being increasingly unreliable. One 
review of the field a decade ago raised questions about the re- 
liability of observational epidemiologic studies when quan- 
tifying the health effects of important, but subtle, risk factors 
such as second-hand smoke, air pollution, and diet (2). 

Scientific evidence is strengthened when important find- 
ings arc replicated by multiple independent investigators 
using independent data, analytical methods, laboratories, 
and instruments. Replication, as described here, has long 
been the standard in the biologic and physical sciences 
and is of critical importance in epidemiologic studies, par- 
ticularly when they can impact broad policy or regulatory 
decisions. Because of the time and expense involved with 
epidemiologic studies, many are often not fully replicable. 


I 

and policy decisions must be made wicli the evidence at -3 
hand. cS 

An attainable minimum standard is reproducibility, where g 
independent investigators subject the original data to their i 
own analyses and interpretations. Reproducibility calls for 
data sets and software to be made available for 1) verifying 
published findings, 2) conducting alternative analyses of the 
same data, 3) eliminating uninformed criticisms that do not 
stand up to existing data, and 4) expediting the interchange 
of ideas among investigators. Ultimately, all scientific evi- 
dence should be held to the standard of full replication and 
the confirmation of important findings by independent in- 
vestigators. However, the desire to quantify small health 
effects and the significant weight placed on epidemiologic 
findings in the policy-making process create a need for epi- 
demiologic studies to meet a minimum standard. We pro- 
pD.sc rcproducihilily to he this minimum standard. 

There are a number of new developments that are inten- 
sifying the need for reproducible epidemiologic research. 


Correspondence to Dr. Roger Peng, Department of Biostatistics, Johns Hopkins Bloomberg School of Public Health, 615 North Wolfe Street, 
E3535. Baltimore, MD 21205 (e-mail; rp6ng@jhsph.edu). 
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The signal-to-iioise ratio in today's epidemiologic studies 
tends to be smaller than it was in decades past simply 
because much of the “low-hanging fruit” has been picked. 
Factors with large relative risks, such as smoking, so- 
cioeconomic status, family historj', and obesity, am well 
established for major diseases. The targets of current inves- 
tigations tend to have smaller relative lisks that are more 
easily confounded, For example, in air pollution epidemiol- 
ogy, the national relative risk of increased mortality is esti- 
mated to be 1 .tK).^ per 1 0 parts per billion of 24-hour ozone. 
Remarkably, an integrated analysis of mortaiity in 95 met- 
ropolitan areas can detect this signal, which translates into 
thousands of excess deaths per year given the universality 
of ozone exposure (3). Nevertheless, the potential for un- 
explained confounding cannot be denied for such a small 
relative risk (4. 5). 

The explosion of new biologic measurements presents ex- 
citing opportunities for epidemiologic studies. We can now 
quantify DNA sequences, single nucleotide polymorphisms, 
and gene and protein expression. We can image the structure 
and function of the brain and other organs. We can quantity 
diet with lengthy dietary-recdl questiounaires and can quan- 
tify disease symptoms and health conditions with multiitem 
instruments. However, because the data tire inherently com- 
plex and high dimensional, there is an incrca,scd potential for 
identifying spurious associations between their components 
and risk factors or health outcomes (6). 

The widespread availability of statistical and computing 
technology is yet another factor contributing to the potential 
for false positive epidemiologic iiudings. It is now easy for 
a researcher to routinely engage in sophisticated optimiza- 
tions across a large number of models and/or variables to 
identify associations that iire of potential scientific interest. 
Even with a single risk factor and a single response, it is 
standard practice to con.sider a potentially large number of 
models in an effort to adjust for differences among the ex- 
posed and the unexposexl, As the number of covariables 
measured increases, so do the degrees of freedom for influ- 
encing the asstKiation between the risk factor and outcome 
and for identifying subgroups in which the association is 
particularly .strong. 

The developments identified above also have the potential 
to increase the power and precision of epidemiologic re- 
search by enhancing our understanding of disease mecha- 
nisms and leading to studies with more targeted hypotheses. 
Modem computing makes possible the organization and 
analysis of large databases, so that w'e can look farther 
and wider for systematic patterns indicating the health ef- 
fects of various risk factors. I'he reproducibility of epide- 
miologic findings from current and future studies will be 
crucial to prt^viding the substance for informed debate re- 
garding policies affecting the public’s healtli, 

DEFINING REPRODUCIBLE RESEARCH 

Repmducihilily is a minimum step that can he taken in 
any study. In the context of epidemiology, a study is repro- 
ducible when it satisfies the criteria in table I, adapted from 
the paper by Schwab et al. (7) and othem. We illustrate 
reproducibility requirements separately for each of the fol- 
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TABLE 1. Criteria for reproducible epidemiologic research 


Research 

componeht 

Requirement 

Data 

Analytical data set is available. 

Methods 

Computer code underlying figures, tables, 
and other principal results is made available 
in a human-readable form. In addition, the 
software environment necessary to execute 
that code is available. 

Documentation 

Adequate documentation of the computer 
code, software environment, and analytical 
data set is available to enable others to 
repeat the analyses and to conduct other 
similar ones. 

Distribution 

Standard methods of distribution are used for 
others to access the software, data, and 
documentation, 


lowing research components: data, mctliods. documenta- 
tion. and distribution. 

Epidemiologic data sets can be dilficult to define because 
of the complexity of the data collection and the preproces.s- 
ing. In addition, a publi.shcd finding may use only a subset 
of the data collected for a particular study, the other parts be- 
ing irrelevimt to the reproducibility of the finding. We there- 
fore separate the data into ‘'analytical” data, which serve as 
the input to a .statistical analy.sis program U) produce the 
results found in the table/figure supporting the paper’s con- 
clusions, and “measured” data, which con.sist of all data tind 
functions thereof that are used to create the analytical data, 
whether or not they are pmt of the analytical data. This 
classification is crude and far trom ideal, hut it strikes a com- 
promise between those data that are necessary tor reproduc- 
tion and those that may be of secondary interest. We propose 
as a first requirement that the analytical data set be made 
available to olbers for reproducing results. 

With the increased use of advanced statistical methodol- 
ogy and larger data sets, analy.ses today are almost alw'ays 
implemented on a computer. Given that, the simplest way to 
reproduce the statistical methods is to examine the computer 
code or instructions for the analysis. WlriJe some analyses 
may be con.sidcred too rudimeuLary to warrant publishing 
computer code, most stati.stical softw'are routines, for exam- 
ple, contain many options that need to be set hy the user. 
Since it is not alw'ays cletir from tire outset which options 
can have an impact on the numerical results, this informa- 
tion can be critical for reproducing scientific findings, par- 
ticularly when investigating small relative risks (8, 9). 



REPRODUCIBILrrv OF CURRENT EPIDEMIOLOGIC 
RESEARCH 

To measure the reproducibility of recent epidemiologic 
research, we review-ed 90 epidemiology articles from the 
American Journal of Epidemiology aud the Journal of 
the American Medical Association. We selected every aiti- 
cle published in 2005 in the time period between January 
and the time at which w’e conducted the review (May), We 
developed a questionnaire to collect information relevant to 
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TABLE 2. Results from examining the epidemiologic 
IKerature; articles from the Amertcan Journal of Epidemiology 
and the Journal of the American Medical Association published 
between January 2005 and May 2005 



No. of papers 

Total papers collected 

90 

Observational studies 

69 

Cross-sectional 

20 

Case-control 

20 

Cohort 

29 

Source of outcome data 

Original study 

31 

Ongoing study 

29 

Government 

8 

Other 

1 

Statistical analysis implementation 

Not reported 

21 

By hand 

0 

Use of software package 

48 

Method of processing measured data 

Not reported 

43 

By hand 

1 

Use of software package 

13 

Outcome data reported to be available 

0 

Exposure data reported to be available 

1 

Code for statistical analysis available 

0 

Code for processing measured data available 

0 


reproducibility that is available at liltp://w\vw',biosLaL.jtisph. 
eduZ-rpeng/reproducible/surv'eyA Some of the articles se- 
lected during tills time period were excluded on the basis 
of the criteria developed in the questionnaire. 

We focused our review of an arlieie on the abstract’s 
conduding statement summarizing the main scientific find- 
ings. Each stalcnscnt contained inforntation about an out- 
come and an exposure vatiable or risk factor, as well as 
potential confonnders and/or effect modifiers. For a given 
article, we first determined the type of study: randomized 
trial, methodology, literature review or meta-anaiysis. or 
tmginal observational study. The survey addressed only 
the last category, as it forms the bulk of epidemiologic re- 
seai'ch. Articles describing other types of studies were ex- 
cluded. Given an observational study. w‘e determined the 
study design, the primary outcome and exposure variables, 
and the tablc.s nr figures providing the evidence supporting 
the concluding statement. We also recorded details of how 
the statistical analysis was implemented and whether or not 
data were reported to be available. The data availability' was 
determined separately for the primary outcome, the expo- 
sure. any potential confounders, and any elTect modifiers, 
since it is commonly the ca.se that the different variables 
have different sources. 

The results of our survey are summarized in table 2. In 
total, we examined 90 articles. 69 of which were observa- 


tional studies and had either a cross-sectional, case-control, 
or cohort design; We focus only on these 69 articles. For 84 
percent of the articles, the data for tlie outcome and expo- 
sure came from original studic.s or from large ongoing stud- 
ies. None of the articles reported that the outcome data were 
available to others, either from the authors or from a third 
party. In only one study were the exposure data available. 
However, in this study. ‘Time” was the relevant exposure 
variable, w'ith the .study examining trends in cardiovascular 
risk factors through a scries of surveys. It should be noted 
that we did not attempt to contact the autliors and to request 
die analytical data and computer code used for dieir pub- 
lished analyses. Had the authors been contacted, it is not 
known how many w'ould have been willing or able to pro- 
vide the data and code. 

Thirty percent of the articles did not report how the sta- 
tistical analyses w'ere implemented, while the remaining 70 
percent reported using a specific software package, Neither 
the software for the sLatislieul analy.sis nor the software for O 

processing the measured data into analytical data was re- § 

ported to be available. Of the articles w'here measured data g 

required processing, 93 percent did not report how- this pro- ^ 

cessing v^'as implemented. =r 

i 

METHODS FOR REPRODUCIBLE RESEARCH b 

S' 

Articles printed in journals ai'c still the primary means by S. 

which scientific results arc presented. How'cver, rcprixluc- ^ 

ible research as defined in the previous section requires that & 

arrangements be made for the distribution of analytical data b 

and methods. While journals typically govern the distribu- “ 

Lion of the scicntilic findings, the task of distributing data ^ 

and methods is relegated to the authors, g 

Today, the World Wide Web is tlie most convenient me- j 

dium for distributing information to other researchers and is ^ 

already playing a cenci'aJ role in the implcmctuation of re- g. 

producible research. Many journals now hav'c websites that g 

can host supplementary materials for published anieles, •5 

such as data that can be downloaded along with computer 
code for reproducing specific anah'ses. In addition, more o 
detailed explanations of methods and complementary fig- ^ 
ures can be provided to the reader who intends to reproduce 
the published findings and to conduct competing analyses of 
the same data set. 


LITERATE PROGRAMMING 

The practice of posting data and code on either pereonai 
or journal websites is a signilicanl first .step. While making 
data and code available is certainly necessaiy, it is typically 
insufficient for othei's fo reproduce results. An author must 
additionally provide details about how tlie code is linked to 
the data and which code sections are applied to which data. 

A compendium is an article linked together with the data 
and code necessary for prtxlucing all of the results in the 
article (10, 1 i ). The tools for constructing a compendium are 
modeled on the idea of literate prograrmninn. a phrase coined 
by Donald Knuth (12) and a concept extended by many 
others (13). A literate program combines a documentation 
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language with a programming language to produce an over- 
all program that is seif-documenting and more “literate." 
Knulh’s original WEB system for writing literate programs 
combined the ThX docuracnlation language with the Paseal 
programming language. In a literate program, one weaves 
tlie text and code to produce a human-readable document 
and tangles the code to produce a machine-executable file. 
The advantage of the literate programming approach is that 
the code and text can provide a running commentary on each 
other. 

The specific details of how a compendium is created 
depend on the computing environment and programming 
languages used by the author. Gentleman and Temple Lang 
(10) propose using the R software environment (15) coupled 
with the L'^TpX document-formatting language. The gen- 
eral idea of a compendium is not tied to any one software 
package but, rather, the ideas of literate programming are 
easily applied to the documentation and programming lan- 
guage with which the researcher is familiar, 

OPEN RESEARCH DATA LICENSES 

It is understandable iliat authors do not make their re- 
search data available, if onty because once data are released, 
there is little control over how the data will be used (16). 
A regime whereby partial rights to research data could he 
granted would allow .some flexibility for authors to make 
data available without giving up complete control. For re- 
producible research (o become the standard in epidemiol- 
ogy, limited access to data is a necessity. 

We propose dirfercnl classes of data licenses that pro- 
\'ide partial rights to research data under prespecified 
conditions. In developing these classes, we boitow from 
standards created by the Creative Commons project (liitp:// 
crcativccommons.org), an organization devoted to creating 
licenses that provide partial rights to literary works. These 
ideas have also been discussed in the software community, 
where “open source” licenses are commonly used to provide 
partial rights to software products (e.g.. the Open Source 
Initiativ'e at httpii'/opensource.org/). 

The following list defines four possible cla.s.ses of data 
iicerises in order of increa.sing restrictivencss. We choose 
not to use precise legal definitions but rather outline the 
basic ideas. 

1. FuU access. Tiie data can he used for any purpose. 

2. Attribution. The data can be used for any purpose so 
long as die audiors are cited (a specific citation should 
be provided). 

3. Share alike. The data can be used to produce new find- 
ings or results. Any modifications to the data, includ- 
ing transformations, additions, or linkages to other data, 
which aie used to produce die new findings, must be 
made available under the same terras. 

4. Reproducrioti. The data can be used for the purpose of 
reproducing the re.sults in the tts.s(x;ialcd published 
article or for commenting on those results via a letter 
to the editor. No original findings based on the data may 
be published without explicit permission from the orig- 
inal investigators in a separate agreement. 


Licenses providing partial rights to data can benefit both 
the donor and the recipient. The recipient obtains access to 
the data and an explicit understanding of the rights granted 
to him or her. The donor meets data disclosure obligations 
(from either granting agencies or journals) and is provided 
some mea.sure of coniroi o\'ef otliers' use of the data in au 
undesirable manner. In addition, the donor is relieved of 
having to negotiate potentially numerous requests for the 
data set. With the benefits also come the costs of such a li- 
censing rogime. The recipient must accept limiuitions to the 
data set by the donor, wliile the donor must initially invest 
time to arrange for data sharing and risks agreement viola- 
tions by those using the data. 


REPRODUCIBLE RESEARCH IN AIR POLLUTION AND 
HEALTH: A CASE STUDY 

We demonstrate one implcmenlalion of reproducible re- ° 

search with a large observational study of the health effects § 

of air pollution, the National Morbidity, Mortality, and Air g 

Pollution Study (NMMAPS), NMMAPS is a national time- ^ 

series study on the shojx-tenn health effects of air pollution, =?■ 

the goals of which are to 1) integrate multiple government 3 

databases that contain information on population health, ■§• 

ambient air pollution levels, weather variables, andsiKioeeo- ° 

nomic I'ariables for air pollution epidemiology; 2) dei'elop ^ 

statistical methods and computational tools for analyz- o 

ing and inteipreting the resulting database; and 3) estimate 3 

the short-term elTcels of air pollution on mortality' and its w 

uncertainty for the largest US metropolitan areas and for J 

several geograpliic regions (17, 18), ^ 

Because of the regulatory context, quantification of air « 

pollution risks i.s controversial. The assessments of risks m 

arc pait of a highly charged policy debate and. eon.scquenlly, § 

statistical methods and data sources are snbjecf to intense j 

scrutiny by other scientists and interested patties. A recent ^ 

review of the epidemiologic evidence on the health effects SJ 

of fine panicles described this debate (19), whieh will likely 
be revisited now that the Environmental Protection Agency 'f.j 
(20) ha.s pntmuigaLed its latest daily and annual standards 2 
for pinticulate matter. In the last few years, NMMAPS and 
several other large epidemiologic studies (2 1-23) ha%'e been 
a part of this policy debate (5). 

A.s a first step toward developing higher standards of 
reproducibility, we created the Internet Health and Air Pol- 
lution Surveillance Sy'stera for disseminating the entire 
NMMAPS database and the software for implementing all 
of our statistical analyses (http://www.ihapss.jh.sph.edu/). 

Other scientists can fully reproduce our results, apply our 
methodology to their fwn data, or apply their methodology 
to the NMMAPS database. One of the goals of our approach 
is to raise the level of scientific debate by making all of our 
methods publicly available and to create new tools and 
standards that encourage others to do the same. 

In addition to the Internet Health and Air Pollution Sur- 
veillance System website, we have created a compendium 
for a recent publication, “Seasonal Analy'ses of Air Pollution 
and Mortality in 100 US Cities” (24), which contains an 
analysis of the seasonal and regional variability of the health 
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TABLE 3. Making resuKs from the National Morbidity, 
Mortality, and Air Pollution Study reproducible” 


Research 

component 

What we have done 

Data 

The entire NMMAPSf database is available to 
the public via the iHAPSSf website and the 
NMMAPS data package for R; the data are 
available under a “full access” class of 
license. 

Methods 

A full compendium written in L*TeX and R is 
available for download. 

Documentation 

We have outlined our data-processing pipeline 
on the iHAPSS wobsite, and papers/ 
technical reports are available for download. 

Distribution 

We use the World Wide Web to disseminate 
our data and software. 


’’’ Details at http://wvw.biostat.jhsph.edu/-rpeng/reproducible/. 
j NMMAPS, National Morbidity, Mortality, and Air Pollution Study; 
iHAPSS, Internet Health and Air Pollution Surveillance System. 


cffect.s of paiticiilatc matter. To allow othert; to reproduce 
our findings, we have developed a simple wehpage that 
contains links to all of the data and computer etKlc for 
generating the figures and tables in the article. The compen- 
dium is w'dlten with die literate programming techniques 
described in the previous section, an excerpt of which is 
shown in the Appendix. Readers can inspect the eodc for 
producing the results and for creating the tables/figures, as 
well as download the code and data to their own computers 
to run other analyses or produce different figures. A sum- 
mary of our efforts can be found in table 3. and wc have 
posted complete infomiation about the data and mcthod.s 
used in this paper at http://www.hiostat.jhsph.edn/-rpeng/ 
reproducible/. In a recent study of fine particulate matter 
and hospitalizations among the elderly (25). we have ap- 
plied the same principles of reproducibility and have posted 
information at http://w’ww.biostat.jhsph.cdLi/MCAPS/. 

DISCUSSION 

In this article, we have proposed that reproducible re- 
search be the minimum standard in disseminating epidemi- 
ologic findings and have demonstrated the possibilities with 
a large ongoing study of air pollution and health. The policy 
implications of epidemiologic studies coupled with the in- 
vestigation of smaller targets, the increasing use of complex 
databases, and the application of sophisticated statistical 
modeling can lead to research that is subject to intense 
scrutiny. The reproducibility of principal findings can foster 
rational discussions regarding The evidence in the data and 
serve as a bulwark against uninformed criticism. 

The standard of reproducibility addresses some critical 
scientific issuc.s, but its reach is stili limited, In order to 
identify the issues that reproducibility can address, we must 
first agree on a model of the research process itself. One can 
think of an epidemiologic study as a .sequence of stages. 
Starting from the generation of the data by natural or otlier 
processes, to tlie collection of these data, to data process- 
ing and analysis, and then to the reporting of results. Prior to 


die generation of the data, one might also include the for- 
mulation of the hypotheses and the design of the study. 

Reproducibility becomes meaningful in die stages of 
a study following the data collection. The processing of 
the data, as w'ell as the analysts and sub.sequeut presentation, 
can all be inspected if the research is reproducible. Beyond 
checking for statistical and programming errors, one can 
evaluate the sensitivity of the findings to certain modeling 
choices. By having the computer code used to process and 
analyze the data, others can obtain useful infomiation re- 
garding the many important choices made as part of the 
study. 

Among the issues that cannot be addressed by reproduc- 
ible research are those arising from stages of the research 
process prior to the data collection stage. Questions about 
the study design, the selection of subjects, the handling of 
nonrespondems, and many other facets of a smdy cannot he 
re.solved with the analytical data alone. Similarly, it may not 
be possible to exiunine all rclevam modeling choices, par- ^ 

ticulaiiy those involving variables for which no data were § 

originally collected. Howev'er, when w'e discuss these types g 
of issues, we are moving closer to calling for full replication. ^ 

If a particular study is fully replicable, then all aspects of the g- 
original study can be adjusted or modified. Clearly, full 3 
replication rcTnains the uilimalc standard by which we eval- 
uaie scientific evidence. § 

Data availability is the first and foremost challenge to 
reproducible epidemiologic research. Although this problem o 

is not unique to epidemiology (26), being observ'atioiial, 3 
evidence from epidemiologic ,sludic.s is more often open to w 
diftering interjvretations. It is exactly in such a circumstance J 
diat work needs to be open and reproducible. We have pro- o- 
posed a framework of ^‘partial rights” to research data that “ 
would modulate the all-or-nothing scenario that exists today. « 

One impediment to making data available is preserving § 

confidentiality. Health data are often obtained by making m 

promises to individuals (either directly or thixiugh an inter- S’ 
mediary) that confidential information about those individ- ^ 
uals will not be released to the public. Under our definition, 
it would seem impossible to simultaueously honor those 
promises and make one’s research rcpn)duciblc. How'cvcr, 2 
while the measured data often must be kept confidential, it 
may still be possible to provide summary stati.stics of the 
data upon which the analysis is based. For example, with 
limc-.scrics studies of air pollution and mortality, the indi- 
vidual mortality data are confidential, but the time series of 
aggregate counts for each county ciui be made available for 
large enough counties. Since the analysis is based entirely 
on those aggregate values, there is no need to release the 
individual-level data. This is a limited example, and al- 
though there is active discussion in the literature about dis- 
closure limitation methods (27), the issue of releasing data 
for the purposes of reproducing scientific findings is in need 
of serious discus.sion. 

The literature review' seiwed both to assess tlie state of 
reproducibility in the epidemiologic lileraiurc and to provide 
a '‘checklist” for producing a reproducible study. In addition 
to data availability, we identified a number of additional 
problems preventing epidemiologic findings from being 
reproduced. The sparse reporting of analytical methods 
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and the lack of compurer code describing those methods ai'e 
of concern. We have demonstrated how to use literate pro- 
gramming techniques to produce a reproducible document 
and the Web for distributing data and software. The repro- 
ducibility of the document is ensured by the use of tools that 
allow text and code to be iiitenningled to form a common 
source for the finished paper. In general, programming 
languages and statistical packages tend to change, and we 
do not presume that there is a single "'best'’ emironment. 
Rather. \vc describe the general concept of literate program- 
ming and highlight some specific tools that are available for 
encouraging such practice. 

The call for reproducible research has already been ech- 
oed in other fields where computation and complex statisti- 
cal methodology are critical for obtaining substantive results 
(7, 10. II, 28-3 1 ). Biologists have made enormous progress 
toward integrating databases, sharing software, and making 
their analyses leproducible. Journals such as Science and 
Nature require deposiliou of biologic data into public re- 
positories at the time of publication, and organizations such 
as the Microarray Gene Expression Data Society have de- 
veloped rigorous stimeiards for the reporting of micioarray 
data (32). The Inter-University Consortium for Political and 
Social Research is a vast repositoiy for social science data, 
providing archiving rc.sourec.s as well as standardization of 
data sets for a number of software environments. Social 
science inve-stigatore intent on making their research repro- 
ducible have a clear resource for sharing their data. 

In addition to various field-specific efforts, the US Na- 
tional Institutes of Health now requires many of its grantcc.s 
to implernent a dafa-sharing policy for any research spon- 
sored by the Institutes. Even more broadly, the federal 
government, via the Shelby Amendment of 1999 and the 
subsequent revision to the Office of Management and Bud- 
get Circular Al 10, requires data from any federally spon- 
sored research to be made available upon request if the data 
were used in "developing an agency action that has the 
force and effect of law” (33, p. 220). It is not yet known 
what the full impact of either of these polieies will be on the 
reproducibility of all biomedical research. 

In the absence of full replication, reproducibility should 
become the minimum standard for epidemiologic research. 
In pai'ticular, studies with potential policy impact should be 
made reproducible to allow others to verify published find- 
ings and to conduct alternative analyses of the data. Wc have 
demonstrated thn^ugb our case study that the standaixl of 
reproducibility can he achieved and have proposed a frame- 
work in which the results can be disseminated, The apparent 
unreliability of epidemiologic investigations predicted 10 
years ago can be thwarted today by adopting new standards 
and embracing a more open research enGronment. 
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APPENDIX 
Sweave Example 

I’he following i.s taken from a vignette for reproducing the results reported in tlie paper by Peng et ai. (24). The document 
is written in the L'^'T^X document-formatting language, and the portions between the <<>>— and @ symbols arc written in 
the R language. 

The national average estimates of the overall and seasonal short-term effects of \PMTen\ 
on mortality for lags 0 , 1 , and 2 are stimmarized in Table 2 . These estimates were obtained by 
pooling the ci ty-specific maximum likelihood estim.ates from the main effect and pollutant- 
season interaction models according to the hierarchical normal model. 

<<natiDnalAverageEstimates , results=tex, echo=£alse>>= 

Seasons <- c ( "Winter" , " Spring" , " Surroner ", ” Fall " , "All Seasons " ) 

Lags <- paste ( "Lag ", 0:2); exclude <- c ( " hono ", " anch" ) 

## Load non-seasonal estimates 

load (file .path ( " results " , " city-specific-est . pmlO . rda " ) ) 

results <- 1 apply ( results , function (x) x [setdif f (names (x) , exclude) ] ) 

## Pool estimates 

betacovTotal <- lapply (results, extraetBetaCov, pollutant —poll) 
pooledTotal <- iapply (betacovTotal , poolCoef) 

## Load seasonal estimates 

load (file . path ( " results " , " seasonal . factor! . lag . 012 . pmlO . rda ” ) ) 
results <- lapply (results , function (x) x [setdif f (names (x) , exclude) ] ) 

## Pool estimates by season 

pooledSeas <- lapply{results, coefSeasonal , pollutant=poll , 
method=method ) 

pooled <- lapply [ seq { along — pooledSeas 1 , functionli) { 
m <- rbind (pooledSeas [ [i ] ] , pooledTotal [ [ i ] ] ) 
rownames (m) <- Seasons 
m 

}) 

@ 
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“Deming, Data and Observational Studies,”: 
Article by S. Stanley Young and Alan Karr 


Deming, data and 
observational studies 

A process out of control and needing fixing 


"Any claim coming from an observational study is most Likely to be wrong." Startling, but true. Coffee causes 
pancreatic cancer. Type A personality causes heart attacks. Trans-fat is a killer. Women who eat breakfast cereal give 
birth to more boys. All these claims come from observational studies; yet when the studies are carefully examined, 
the claimed Links appear to be incorrect. What is going wrong? Some have suggested that the scientific method is 
failing, that nature itself is playing tricks on us. But it is our way of studying nature that is broken and that urgently 
needs mending, say S. Stanley Young and Alan Karr; and they propose a strategy to fix it. 


Science works by experiments that car be repeated: 
when they are repeated, they must give the same 
answer. If an experiment does not replicate, 
something has gone wrong, In a large branch of 
science the experiments are observational studies: 
we look at people who eat certain foods, or take 
certain drugs, or live certain lifestyles, and we 
seem to find that they suffer more from certain 


diseases or are cured of those diseases, or - as 
with women who eat more breakfast cereal - that 
more of their children are hoys. The more startling 
the claim, the better. These results are published 
in peer-reviewed journals, and frequently make 
news headlines as well. They seem solid. They 
are based on obsenvati'on, on scientific method, 
and on statistics. But something is going wrong. 


There is now enough evidence to say what many 
have Long thought; that any claim coming from 
an observational study is most likoly to ho wrong 
- wrong in the sense that it will not replicate if 
tested rigorously. 

As Long ago as 1988’’^ it was noted that 
there were contradicted results for case-control 
studies in 56 different topic areas, of which 


Table i. We have found 12 papers in which claims coming from observationalstudies were tested in randomisec! clinical trials. Many of the trials are cruite large. In most 
of the obseivationai sttidies multiple daiins were tested, often in factorial designs, e.g, vitamin D and caieiKm individually and together along with a placebo group. Note 
that nano of the daimr replicated in the diW'Ction claimed in the observational sUioies and that there was statistical significance 'in the opposite direction five times 


ID no. 

P05. 

,/v'eg. 

No. ofcbiws 

Treatment{s) 

Reference 

1 

Q 

1 

3 

Vit £, beta-carotene 

NEJM 1994; 330 : 1029-1035 

2 

0 

3 

4 

Hormone Replacement Ther. 

JAMA 2003; 289 : 2651-2662, 2663-2672, 2673-2684 

3 

0 

1 

2 

Vit E, beta-carotene 

JNCI2QQ5: 97: 481-488 

4 

0 

0 

3 

VitE 

JAMA 2005; 293 ; 1338-1347 

5 

0 

0 

3 

Low Fat 

JAMA. 2006; 295 : 655-666 

6 

0 

0 

3 

Vit D, Calcium 

NEJM 2006; 354 : 669-683 

7 

0 

0 

2 

Folic acid, Vit B6, B12 

NEJM 2006; 354 ; 2764-2772 

8 

0 

0 

2 

Low Fat 

JAMA 2007; 298: 289-298 

9 

0 

0 

12 

Vit C, Vit E, beta-carotene 

Arch Intern Med 2007; 167; 1610-1618 

10 

0 

0 

12 

Vit C, Vit E 

JAMA 2008; 300 : 2123-2333 

11 

0 

0 

3 

Vit E, Selenium 

JAMA 2009; 301 : 39-51 

12 

0 

0 

3 

HRT + Vitamins 

JAMA 2002; 288: 2431-2440 

Totals 

0 

5 

52 
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cancer and things that cause it or cure it were 
by far the most frequent. An average of 2.4 
studies supported each association - and an 
average of 2.3 studies did not support it. For 
example, three studies supported an association 
between the anti-depressant drug reserpine and 
breast cancer, and eight did not. It was asserted' 
that "much of the disagreement may occur 
because a set of rigorous scientific principles 
has not yet been accepted to guide the design 
or interpretation of case-control research" 
Problems extend to essentially all observational 
studies. Little progress has been made to adopt 
rigorous scientific principles. Some journal 
article titles give a flavour of the sentiments: 
"Epidemiology faces its limits", "Is it time to call 
it a day?", "Have we learned from our mistakes, 
or are we doomed to compound them?" In the 
popular press, an article by Oonah Lehrer in 
the New Yorker^ bore the subheading "Is there 
something wrong with the scientific method?" 
and seemed to imply that replicability was no 
longer occurring; it concluded with Cbe phrase: 


"When the experiments are done, we stid have 
tc! choose what to believe." No. In the Lehrer 
example the motivating finding was wrong and 
therefore should not be expected to replicate. 

It may not be appredated how often 
observational claims fail to replicate. In a small 
sample in 2005', of 49 claims coming from highly 
dted studies, 14 either failed to replicate entirely 
or the magnitude of the claimed effect was 
greatly reduced (a tegressiori to the mean). Six 
of these 49 studies were observational studies, 
and in these six, in effect, randomly chosen 
observational studies, five failed to replicate. This 
last is an 83% failure rate. In an ideal world in 
which welt-studied questions are addressed and 
statistical issues are accounted fcr properly, few 
statistically significant claims are false positives. 
Reality for observational studies is quite different. 

We ourselves carried out an informal but 
comprehensive accounting of 12 randomised 
clinical trials that tested observational claims 
- see Table 1. The 12 clinical trials tested 52 
observational claims. They all confirmed no 


claims in the direction of the observational 
claims. We repeat that figure; 0 out of 52. To put 
it another way, 100% of the observational claims 
failed to replicate. In fact, five claims (9.6%) are 
statistically significant in the clinical trials in the 
opposite direction to the observational claim. To 
us, a false discovery rate of over 80% is patent 
evidence Chat the observational study process 
is not in control. The problem, which has been 
recognised at least since 1988, is systemic. 


The cause of it all 

The cause is elusive and can be considered 
both technically and operationally. Individual 
researchers, the vjorkers, respond rationally to 
incentives by publishing papers in peer-reviewed 
journals and securing funding for their research. 
The quality of their papers is judged by funding 
agencies and journal editors, the important 
managers of the observational study production 
system. We can turn here to statistician W. 
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Edwards Oeming^ the most visionary innovator 
ever on quality control and the man who 
transformed first Japanese car manufacturing 
then manufacturing quality control worldwide 
(see Box 1). Deming said; "The worker is 
not the problem. The problem is at the topi 
Management!" To Deming, blarning the workers 
- individual researchers ' is as incorrect as it is 
useless. Bringing the system under control is the 
responsibility of those managing it. 

What is needed to fix the system? 
Among Darning's famous ''Fourteen Points 
far Management", the third is most directly 
relevant: cease dependence on inspection to 
achieve quality. Every successful company 
today relies on control of the process; they 
do not wait until the end of the process and 
then throw away bad product. That would 
be product control, not process control. It is 
wasteful to make something, then inspect 
and throw away the bad product. Instead, 
every step of the process is monitored and 
controlled, so that had product is net made. 
The "observational studies industry" must 
build a good product: journal editors cannot 
inspect bad product out at the publication 
stage, let alone the replication stage, if the 
processes are controlled by management, the 
products can be sound studies. Control of the 
processes is feasible, and requires attention to 
the incentives, publications and grants. First we 
examine three of the main technical difficulties 
with observational studies; Multiple testing, 
bias, and multiple modelling. 


some of those questions will by chance came up 
positive. Producing at least one false positive 
becomes a near certainty unless the data analysis 
accounts for the multiple questions. Figure 1, 
from the excellent website xkcd.com, briiiiantiy 
explains the basic problem. The "females eating 
cereal leads to more boy babies" claim translated 
the cartoon example into real life. The claim 
appeared in the Proceedings of the Royal Society, 
Senes B. It makes essentially no biological sense, 
as for humans the Y chromosome controls gender 
and comes from the male parent. The data set 
consisted of the gender of children of 740 mothers 
along with the results of a food quesb'ornaire. 
not of breakfast cereal alone but of 133 different 
food items - compared to only 20 colours of jelly 
beans. Breakfast cereal during the second time 
period at issue was one of the few foods of the 
133 to give a positive. Wc reanal.ysed the data^ 


I 

I 

3 ! _____ 
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with 262 t-tests, and concluded that the result 
was easily explained as pure chance. 

For those who want more than cartoons, 
a simple web simulation' is convincing that 
multiple testing needs to be controlled. 
Although many workers who are thought leaders 
of researchers doing observational studies argue 
against any correction of the analysis for multiple 
testing®, managers can require that authors deal 
with multiple testing. 

Bfas 

Whereas multiple testing is random error, bias 
is systematic error. To illustrate it, consider 
channelling, where doctors steer certain patients 
to particular treatments. For example, doctors 
directed HIV patients at high cardiovascular 
risk to a particular HIV treatment, abacavir, and 
lower-risk patients to other drugs, preventing 
a simple assessment of abacavir compared to 
other treatments. An analysis that did not 
correct for this bias unfairly penalised the 
abacavir, since its patients were more high-risk 
so more of them had heart attacks (Figure 2). 
Another problem is that covariate adjustment is 
widely used, but is vulnerable to manipulation 
and is well known to give unreliable results 
when the treatment groups are not comparable; 
see "Multiple modelling" below. Missing 
factors, unmeasured cenfounders, and loss to 
follow-up can also lead to bias. For example, 
in a study published in Pedintnes’, offspring IQ 
was the issue, yet IQ of the fethers was not 
measured and of the 505 children starting the 
Study, 256 (50.7%) were lost to follow-up. By 
selecting papers with a significant p-vaLue, 
negative studies are selected against - which is 


puhlicati 

on bias (see Box 2). 





Afi' events 


Drug 

% high-risk 
patients 

per 2000 
patient 
years 

1 

Tidovudine 

19.0 

3.43 

2 

stavudine 

18.8 

3-72 

. 3 

lamivudine 

18.6 

3.68 

4 

didanosine 

22.1 

4.47 

5 

abacavir 

27.5 

6.13 


Multiple testing 

False positives do occur, even in an ideal world. 
When many questions arc asked of the same data, 

1 ] 8 i agraficonc© 


Risk 

Figure ?, Events per thousand pationt-years are plotted against estimated risk of a heart attack. RHky patients 
WPP channeited to the HIV drug ABC. abacavir, and thr.se patients had more heart attacks, as shown hy the 
uppermost pniritc-n the graph, Risk-adjusted, ail the drugs appear to be of eqiiai risk. Source: Laocst 373, 1417 ff. 
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Multiple modelling 

This problem is akin to - but less well 
recognised and more poorly understood than - 
multiple testing. For example, consider the use 
of linear regression to adjust the risk levels of 
two treatments to the same background level 
of risk. There can be many covariates, and 
each set of covariates can be in or out of the 
model. With ten covariates, there are over 1000 
possihlo models. Consider a maze as a metaphor 
for modelling (Figure 3). The red line traces the 
correct path out of the maze. The path through 
the maze looks simple, once it is known. 
Returning to a linear regression model, terms 
can be put into and taker out of a regression 
model. Once you get a p-value smaller than 
0.05, the model can be frozen and the model 
selection justified after the fact. It is easy to 
justify each turn. 

The combination of multiple testing and 
multiple modelling can lead to a very large 
search space, as the example of bisphanol A in 
Box 3 shows. Such large search spaces car give 
small, l^lse positive p-values somewhere within 
them. Unfortunately, authors and consumers are 
often [ike a deer caught in the headlights and 
take a smaU. p-value as indicating a real effect 


How can it be fixed? A new, combined 
strategy 

It should be clear by now that more than small- 
scale remedies are needed. The entire system 
of observational studies and the claims that 
are made From them is no Longer functional, 
nor is it fit for purpose. What can be done to 
fix this broken system? There are no principled 


ways in the literature for dealing with model 
selection, so we propose a new, composite 
strategy. Following Deining, it is based not 
upon the workers - the researchers - but on 
the production system managers - the funding 
agencies and the editors of the journals where 
the claims are reported. 

We propose a multi-step strategy to help 
bring observational studies under control (see 
Table 2). The main technical idea is to split the 
data Into two data sets, a modelling data set 
and a holdout data set. The main operational 
idea is to require the journal to accept or reject 
the paper based on an analysis of the modelling 
data set without knowing the results of applying 
the methods used for the modelling set on the 
holdout set and to publish an addendum to the 
paper giving the results of the analysis of the 
holdout set. We now cover the steps, one by one. 

1 The data collection and clean-up should 
he done by a group separate from the 
analysis group. There can be a tempta- 
tion on the part of the analyst to do some 
exploratory data analysis during the data 
clean up. Exploratory analysis could lead 
to model selection bias. 


2 The data cleaning team creates a 
modelling data set and a holdout set and 
gives the modelling data set, less the 
item to be predicted, to the analyst for 
examination. 


Table 2. Steps 0-7 can .be used to help bring the 
□bssT'/ebonai stiitiy process inta controL Cunently 
researchers analysing ubservaimnal data sets are 
under no efrective oversight 

Step Process / Action 

0 Data are made publicly available 

1 Data cleaning and analysis separate 

2 Split sample; A. modelling; and B, 
holdout (testing) 

3 Analysis plan is written, based on 
modelling data only 

4 Written protocol, based on viewing 
predictor variables of A 

5 Analysis of A only data set 

6 Journal accepts paper based on A only 

7 Analysis of B data set gives Addendum 
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3 The statistical analysis plan is written 
based or access to all the modelling data 
except the responsels) to be predicted'^^. 

4 The analyst writes down and files the 
statistical pfotocol. The point is that the 
analysis should not be guided by looking 
at the results of exploratory analysis. It is 
too easy to move predictors into aid out of 
an evolving statistical models. Reconsider 
the maze (Figure 3). Given flexibility, the 
analyst can move the answer around. Such 
flexibility must be prevented. 

5 The analysis is done and the paper written 
(see Box 2). 

6 The journal agrees to accept or reject the 
paper without knowing the results of the 
analysis of the holdout data set. 

7 Once that analysis is done, an addendum 
will be added to the paper using the 
specified analysis on the holdout set. 


A hold-out set of dctfa con be tested 
against cioims; if the test faiis, both 
author and journoi stand to be 
embarrossed 




it the holdout set does not support the original 
claims of the author. Both the author and the 
journal are at present living in a Largely risk- 
free environment. False results may never be 
overturned. The claim that "Type A personality 
causes heart attacks" still lives and took dec- 
ades to be declared invalid. Mo-st who took the 
claim at face value to he true never got the 
word that it is not true. The myth still lives. 
The protocol we suggest would have scotched 
it at birth. 

Before our steps 1-7 begin, there is another 
step to be made. Step 0, making data available, 
provides additional oversight. Note that the 
split-sample strategy can control multiple 
testing and multiple modelling, but not bias. 
Bias can be controlled by setting a threshold of 
effect, say for risk ratio a value of 3 to 4'^, of 
effect to be considered actionable evidence of 
cause and effect. 


What can be done? 

Note that workers have known of problems since 
at least 1988 and have instituted none of the 
steps 0-7 in Table 2. Asking authors voluntarily 
to provide protocol, data and analysis code has 
been very largely ineffective. There is a real 
limit to what an individual can do to improve 
the situation, as most of us are consumers. 
Individuals can write letters to the editor 
saying that without access to data the research 
is largely "trust me” science. The incentives 
need to be charged and that can only come 


from the managers of the process. Managers 
cannot carefully examine each published claim, 
but funding agencies and editors can require 
"reprodiidble research". Reproducible research is 
research where the study protocol, the electronic 
data set used for the paper, and the analysis 
code are all publicly available. Managers can 
also require split-sample analysis strategies and 
other methods to protect against false positives. 
At present, researchers - and, just as important, 
the public at large - are being deceived, and 
are being deceived in the name of science. This 
should not be allowed to continue. 

References 

1. Mayes, L. C., Honvitz, R. I. and 
Feirstein, A. R, (1988) A collection of SC topics 
with contradictory results in case-control research. 
[nternot.ionalJournal af Epidemiology, 17 , 6B0-685. 

2. Feinsteir, A. R. (1988) Scientific standards 
in epidemiologic studies of the menace of daily life. 
Sdence, 242 , 1257-1263, 

3. Lehier. 2. (2010) The truth wears off. iVeiv 
Yorker, December 13th, p. 52. 

4. loanniriis, J. P. A. (2D05) Contradicted 
and initially stronger effects in highly cited dirical 
research. Joumo/ of the Amencon Medicoi Assodotioo, 
294 , 218-228. 

5. Wikipedia (2011) W. Edwards Deming, 
i:dward.s_tiGming 

5. Young, S. S., Bang, H. and Oktay, k. 
(2009) Cereai-induced gender selection? Most likely 
a multiple testirig false positive. Proceedinqs of the 
RoydSodety, Series B. 654, 1211-1212. 

7. Dalial, J. (2011) There must he something 
buried in here somewhere, l.s -p ; //v.'ww . 
jeirydallal . ccjti/LHiiP.-'rnulLtyst . him 

8. Rcjthman, K. J, (1990) No adjustments 
are needed for multiple comparisons. Epidemioto^, 
1, 43-46- 

9. Perers, F. P., !.i, Z., Whyatt, R., Hoepner, 
L, Wang. S., Camann, D. and Rauh. V. (2009) Prenatal 
airborne polycyclic aromatic hydrocarbon exposure 
and child 10 at age 5 years. Pediatrics, 124, el95- 
e202 

10. CLaeser, E. L (2006) Researcher incentives 
and empirical methods. Harvard Institute of Economic 
Research Discussion Paper No. 2122. hr.-p;// 

11. Young, S. S. and Yu. M. (2009) To the 
Editor. Journal of the American Medical Association, 
301, 720-721. 

12. Rubin, D. B. (2007) The design versus tlie 
analysis of observational studies for causal effects; 
Parallels with the design of randomized trials. 
Statistics in Medidne. 26, 20-36. 

13. Temple, R. (1999) Meta-analysis and 
epidemiologic studies in drug development and 
poslmarketing sutveilLance. Journal of the American 
Medical Association, 231, 841-844. 

S. Stanley Young is the Assistant Director uf 
Bioinformatics and Alan Karr is the Director al the 
National Institute of Statistical Sciences. 


120 I sigiTficance 


O 




